Professional Blog

For more from me about work, business, technology, kubernetes, and more, check out my medium profile: https://medium.com/@KarlKFI

Standard Engineering Levels

Software Engineering levels are a mess.

Every company decides what their levels are, what they mean, what they're called, what the pay scales are, and they're all different. But what if they weren't?

Why do we even need levels?

Some companies, like Netflix and Valve, have decided to forgo levels and inflated titles in favor of deliberate flattening. This often goes along with a flattening of organization structure as well. Both decisions have pros and cons. I can't effectively argue for flattening, so I wont waste keystrokes on that (yet). Feel free to provide reasoning for that in the comments. Instead, lets look at why you might want levels...

1. Levels provide a path of advancement, communicating expectations and milestones for career growth.
2. Levels codify seniority, a way to for management to communicate and grant authority.
3. Levels promote fairness of pay, mapping capabilities, responsibilities, and experience to compensation.
4. Levels allow for rationalizing the hiring of higher paid, more experienced employees, without disgruntling existing employees or getting push back from finance.
5. Levels make it easier to scale up by codifying management policies.
6. Levels allow applicants to self-select into a desired band of responsibility and pay, increasing attractiveness of job postings to experienced candidates.
7. Levels provide a reward mechanism for social recognition through titles and implied respect.

When should you institute levels?

Putting aside deliberately flat organizations, many startups and small companies also lack levels, simply because they aren't big enough to need to stratify or just haven't gotten around to it yet. 

Any one of the above reasons to have levels might drive you to start talking about them with your team. In my experience, when company size exceeds Dunbar's number (between 100 and 250)  the social relationships within the company start to break down and you start needing much more process and policy to hold the company together. The next point of inflection comes when a single department exceeds Dunbar's number. So if your engineering department reaches 100+ you might start thinking seriously about levels, if you haven't instituted them already.

The critical point comes when your employees start asking "Why should I care what this person thinks? I don't know them. I don't work with them regularly. I don't know their experience or specialty." Now you might counter that with "You should trust that your company hired smart people." But that's not a very satisfying answer, especially if you don't know their boss either.

Really, it comes down to communication. Levels communicate several things. If you institute levels you communicate those things explicitly so that you don't need to communicate them implicitly or one-on-one any more.

How many levels is enough?

There's no one right answer to this question. As with most engineering questions, the answer is "It depends." 

One of the oldest example of levels is seen in the guild system of artisans or merchants. Guilds helped originate the idea of lifetime progression from apprentice to journeyman to master to grandmaster. These levels allowed guild members to demand recognition and increased compensation from their peers and their customers.

Guilds were eventually largely supplanted by corporations, but the meme of levels hasn't gone away. In some ways, the idea of artisan levels even overlaps somewhat with concepts like The American Dream, which revolves around the availability and achievement of upward social mobility. If each level grants increased compensation and recognition, then advancing through the levels also advances one socially. Of course, engineering levels wont span the same gamut, from pauper to titan of industry, but it does express a subset, perhaps just within the middle and upper middle classes.

Interestingly, most modern leveling rubrics define a lot more than just the four levels used by the guild system. A fascinating resource is levels.fyi. Given that data, it seems the majority of represented companies have between 5 and 9 distinct software engineering levels. For example, Google's has 9 software engineer levels while California has 5 programmer analyst levels. Both of these are popular examples to emulate based on a history of successful operation and logical separations. 

What level are managers?

One thing that's called out in California's levels, that isn't as obvious from Google's, is the qualification of "supervisor", which usually means direct management. Despite the clarity in the official series tho, the implementation of matrix management often confuses the topic. Google's, on the other hand, focuses on technical leadership, with management being a tangential concept. You might have to be at least an T5 (senior) to manage other people, but you might also be an T8 (principal) with no direct reports. 

Having worked under both types of regimes, I think it makes more sense for management to either be an attribute that modifies your level, or a separate leveling series that branches from and overlaps with the independent contributors, rather than trying to fit both into the same leveling series. The biggest reason for this is that management is effectively a completely different skill set than programming. Both roles eventually get into politics and heavy communication at higher tiers, but they often require distinctly different personalities, talents, experience, and goals.

Are levels the same as titles?

Again, it depends. Titles are hard. 

The more interesting question is perhaps whether titles should indicate role or just level. If you're on the Platform Team and you're a Senior Engineer, does that make you a Senior Platform Engineer? What if you're new to platforms but not to engineering?

This complexity has lead to a situation where many tech companies let you make up your own title, or put whatever you want on your business card. This isn't really a solution tho, especially if you're using levels to help communicate. Substituting a user chosen title probably communicates something else, generating confusion rather than alignment and clarity.

My suggestion? Compound titles. Being a "Senior Engineer on the Platform Team" communicates a lot more effectively than "Senior Platform Engineer", unless for some reason you have so many Platform Engineers that they get their own leveling series. Plus, with compound titles, you can easily indicate that you're a manager or supervisor if you don't have a distinct leveling series for that.

Another branch that is sometimes implemented is a separate leveling series for architects. This is more controversial and less common, but as with management, system architecture spanning multiple projects and teams and domains often requires different personalities, talents, experience, and goals than programming or design of individual components and smaller scale systems.

What might standard levels look like?

These levels probably aren't perfect, but they're probably a good place to start from, and iterate on if you decide they don't match your needs exactly.

For reference, guild levels are mapped to every other level. This is because the guild levels are well defined historical ideas that help communicate without needing to define them every time you use them. 

1. Intern
2. Associate (Apprentice)
3. Senior Associate
4. Staff (Journeyman)
5. Senior Staff
6. Principal (Master)
7. Senior Principal
8. Fellow (Grandmaster)
9. Senior Fellow

The target of these levels was originally software engineers, but it turns out they're applicable to a variety of positions, like designers, analysts, hardware engineers, technical sales, etc.

Who is junior?

No one likes being called junior. You could get away with calling an intern junior, but you might as well just call them an intern. It's more descriptive and less derogatory. Likewise, you don't need to call anyone entry if everyone above them is qualified by seniority.

What is an associate?

Retail companies, especially in high fashion, often call their entry level of floor sales people Sales Associates. This almost ubiquitous usage just isn't worth conflicting with. 

An associate is someone who is connected with the organization or has subordinate membership. The connotation of being connected to but not necessarily a part of the organization may not be entirely accurate when used for Full Time Employees (FTEs), but it does at least imply at or near entry level.

If the word itself is contentious, one option is just to leave it out, without affecting the leveling. For example, T2 could be "Software Engineer" while T3 is "Senior Software Engineer".

Who is senior?

Some people really don't like the deflation of the "senior" qualifier common to the tech industry. Senior when discussing age usually means over 65 years, the relative peak of life, rather than something describing 20-somethings. But when you have more than a few levels, it's often hard to find English qualifiers to express seniority. 

Some companies use Senior as the level above staff. Some companies use Senior as the second level above no distinct title. For my proposal, I used Senior as the "tock" to the guild level's "tick". This way Senior gets a distinct meaning as a sub-qualifier and doesn't need to be put on the same scale relative to the primary qualifiers of Associate, Staff, Principal, and Fellow.

Another benefit of making Senior a sub-qualifier is that you sidestep the problem of whether Senior is above or below Staff, which often confuses people.

Who is a lead?

Tech Lead or Lead is often used as a level, but in my experience it tends to imply leading a team of peers. If you have more or less leads than teams then who's leading each team? Lead is too tightly coupled with the organization structure and roles. So it's easier just to leave it out of the levels.

Why not distinguished?

The problem with using "Distinguished" as a level is that it implies that one stands out from the crowd. It's hard to go further than being distinguished. It doesn't make sense for Senior to modify Distinguished either, because they are both adjectives. So either Distinguished needs to be at the top or it needs to be a third modifier above Senior, which puts it too low. What exactly would a Distinguished Associate be? It sounds more important than a Staff member. So that's not going to work.

The other problem that Google ran into is that they had to add more levels as time went on. The top when they instituted levels wasn't enough. For example, they eventually needed to add Senior Fellow to give Jeff Dean a new title in 2013. So putting Distinguished at top makes an artificial cap you might need to expand past.

Why not partner?

I don't want to answer every objection here, but there's definitely some places, like Microsoft, that use Partner as an alternative to Fellow or at least above Principal. I left it out for a few reasons. First, partner is often used to describe external individuals or companies that are being collaborated with for integration or supply or sales, and having it be a title might be too ambiguous. Second, law firms and banks often use Partner to describe partial owners of a private firm, but this conflicts with ownership through public stock and can be confusing when a private company goes public.  

How do I implement this?

Another hard question! I'm gonna call this out of scope of this article, otherwise I'll never finish. ;) But there's definitely more to do that I will leave as an exercise for the reader and maybe explore in a future post:

• What responsibilities does each level have?
• What are the pay scales for each level?
• What experience or knowledge is required for each level?
• How does leveling affect who you report to?
• How do you re-level people when instituting levels?

The Role of the Software Architect

An architect's role is to provide technical vision, coordinate direction across teams/components, and counterbalance short term thinking.

— Karl Isenberg (@KarlKFI) July 7, 2017

What's in a Name?

Collaboration improves when the roles of individual team members are clearly defined and well understood.

— Tammy Erickson April 05, 2012

I recently switched roles, from distributed systems engineer to distributed systems architect. I've been doing professional software development for about ten years. I remember telling my boss five years ago that my five year plan was to be a software architect. Guess that worked out! (Now what!?)

Looking back, it's obvious that I'm much better equipped for the position than I was five years ago. But the role itself also seems to have changed over time, as the tech world has evolved to integrate agile tendencies. So as I approached the new position, I found myself needing to repeatedly answer the same questions. Here are some of my notes.

What is a "Software Architect"?

An architect is an engineer who empowers other engineers as a force multiplier.

An architect focuses on the big picture of the system as a holistic product, rather than owning individual components.

An architect drives consensus between tech leads and product managers, rather than making or dictating critical decisions.

An architect promotes flexibility by identifying potential interface points and presenting engineers with options to reduce the cost of change, balancing short-term tactics and long-term strategy.

An architect is a specialist with strengths in diagramming, documenting, communicating, information architecture, and system design.

What is a "Distributed Systems Architect"?

The title itself doesn't seem to have a commonly accepted definition. So let's build one from similar titles.

A systems architect defines the architecture of computerized systems to fulfill specified requirements. Such definitions include the design and specification of components, component interactions, interfaces, technologies, resources, and dependencies.

— Wikipedia - Systems architect

A distributed system is a model in which components located on networked computers communicate and coordinate their actions by passing messages to achieve a common goal. Significant characteristics of distributed systems include: concurrency of components, lack of a global clock, and independent failure of components.

— Wikipedia - Distributed computing

Therefore, a distributed systems architect is a systems architect who defines the architecture of distributed systems, with special focus on communication and coordination between components.

Increasing Complexity

Whether you call yourself a software architect, a systems architect, or a distributed systems architect, the computing landscape continues to evolve with or without you. A few things specifically have changed in the last decade or two that make these roles harder than they used to be.

1) The demand for high availability and intolerance of down time has caused components that used to be simple to be replaced by distributed systems. Likewise, previously distributed systems became systems of systems. With multi-layer systems it's no longer appropriate to rely on legacy architectural design patterns, like a single API layer or a single communication bus.

2) The demand for agility and larger development teams has mutated service-oriented architecture into microservice architecture. This mode of development pushes even more glue code out of the individual components and into the platform or infrastructure layers, increasing their complexity. Deploying and managing a sufficiently large system of microservices increasingly requires a container or application platform above and beyond the scope of modern infrastructure platforms.

3) The demand for hyperscale (the ability to quickly scale up and down without re-architecting) often calls for the architect to design a dynamic system that considers dramatic change over time normal to system operation. The ability to survive a flash crowd of users (aka the slashdot effect) caused by social media is increasingly required by even the most mundane of business domains. Preparation for this and similar phenomenon happens largely at the system level.

4) The demand for open source software creates proprietary systems that aren't completely controlled by their developers. Unlike proprietary components, where one company is in charge of development, open source components can be both easier to change (with a fork or local modification) and harder to persist change (merging changes upstream often requires politics, persuasion, and modification). Partial control of the architecture of a system is nothing new, but partial control over individual components (specifically their APIs and communication patterns) increases demand for intermediary translation, proxy, and caching services. These ancillary components are often tightly coupled with one component to allow loose coupling with another, but tight coupling in a microservice environment often requires new deployment and management patterns, like sidecars and pods.

Musings

Not all software engineers will or should aspire to be software architects. Architecture is a specialization of the engineering track for people who enjoy abstract thinking, are good at and enjoy documentation and diagramming, have the ability to be political and drive consensus, and have demonstrated a solid grasp on prior art and the competitive landscape.

An architect may or may not also be an engineering manager or tech lead, depending on how much focus on architecture is required or desired. Since being hands off of code tends to erode peer confidence and personal skill set, it is desirable that an architect continue to be partly involved in programming efforts, either through tool, component, or ecosystem development.

The bigger an architecture team gets, the more they tend to lose touch with the every day life of the engineers they're supposed to be supporting. However, the more engineers an architect has to support, the less time they will have to stay hands on with code. So a balance must be struck such that architects retain respect and skill yet also spend enough time on architecture to actually provide the benefit of a higher level perspective. It's a fine balance.

Ontological Evolution

The idea of a distinct architect role has in recent past become somewhat contentious among software engineering agile practitioners. The more senior a thought worker, the less they want to be told what to do. But at the same time, modern development patterns are pushing complexity out of individual components with well defined teams into the between space. Plus each component may have a life of its own: independent release cycles, independent versioning, multiple interface/protocol paradigms, independent motivations, and external contributors. Someone has to align these components and the teams that develop them, to unify them into a single cohesive system. So just as engineering practice has evolved, so must architecture practice evolve from controlling and managing technical vision to leading the direction of its evolution. And since collaboration is easier with clearly defined roles, hopefully this new definition can help point the way.


(Revised on 2017-07-13 to expound on the increasing complexity of systems and the increasing demand for architectural design.)

My Other Computer Is A Datacenter

Managing a datacenter doesn't require a hyperwall, just the right operating system.

ersi hyperwall photo by Kris Krüg on Flicker

Today, your personal computer is local. You own the hardware. The data is in a box under your desk. The processes run on a local operating system you installed.

Even those simple statements, which used to be absolute assertions, are getting fuzzier and fuzzier. You probably have data in "the cloud". You probably have used Google Docs, an application in the browser, running in Google's datacenters. Your email hasn't lived on your machine for a long time.

The next generation of computing, however, runs on multiple machines. Businesses are already doing this, running large datacenters, hosting applications distributed across multiple machines. Increasingly this looks like a new form factor. Increasingly abstractions hide the distribution and present a unified view. Increasingly the big players in the field are talking about their software as presenting a distributed kernel or distributed operating system for applications and services to run on.

What does this look like?

Lets take a look at the most mature of these: Mesos.

Kernel

Mesos is a generic distributed resource manager. Applications written for Mesos can schedule their processes on any number of distributed machines. Those processes might be short lived, like batch jobs, or they might be long lived, like web application servers. Those processes are also isolated, wrapped in containers, unable to negatively impact each other, enabling multiple applications and multiple users to live in harmony in the same shared environment.

If Mesos is a distributed kernel, what are the other operating system services?

Init System

How do you start up long lived applications? How do you restart them if they crash? What's the parent process that hosts other applications? On Mesos, the answer is Marathon: a distributed init system that runs applications in containers. It's simple; it accepts docker containers; and it's low level enough to run other applications, even ones that want to spawn other processes (Mesos tasks).

Storage

Compute resources (cpu, memory, processes) are core to an operating system, but you can't have a full OS without device drivers that talk to hard drives or other storage abstractions. In the distributed Mesos datacenter storage takes many forms. The most familiar of those is HDFS. It looks and acts very much like a traditional file system. It's easy to migrate to, and maintains multiple copies of stored data to survive individual drive failures. But it's not the only player. Need a document store? Hadoop works on Mesos. Need robust key/value storage? Cassandra works on Mesos. Need a SQL database? MySQL (Mysos) runs on Mesos.

Jobs & Time-Based Jobs

Not all applications are long lived. Some are ephemeral. For chaining those jobs together, use Spark. Need to run your jobs at a certain time, regularly? On a linux OS, cron handles those jobs. On a distributed Mesos datacenter, Chronos handles fault-tolerant time-dependent job scheduling. Got MapReduce workloads? YARN (Myriad) works on Mesos too.

Logging

Every app logs differently. You can easily log to one of the storage services, or you can use one of several logging or metrics services. Need a publish/subscribe data stream? Kafka runs on Mesos. Need a data stream backbone? Run Flume in Marathon and pipe the data to Kafka. Need real-time distributed search and analytics engine? ElasticSearch works on Mesos. 

User Space

Not all applications are run by the OS init system. User space applications are often more complicated, run on-demand, and have multiple components and service dependencies. For that, use Kubernetes.

Process Dashboard

The penultimate layer of this new ecosystem is the management of this new form factor. How do you observe something that's bigger than you? You have to step back and simplify. You have to take all the resource consumption data and distributed application process metrics and display them in a readable, browsable, navigable format. You need DCOS, which wraps Mesos to provide just that.

Package Management

Want to install all these newfangled datacenter applications and services? DCOS again comes to the rescue, with its command line package manager and app-store-like Mesosphere Universe.

The datacenter is more powerful than ever, and now you have the tools to take advantage of it.

Discuss on Hacker News

How To: Develop Good Managers

"people don’t quit companies—they quit managers" - Chris Loux

In my anecdotal experience, the above is often true.

Technically, I want the freedom to determine the best solution to a problem, even the freedom to figure out what the problem actually is. But career-wise, I want a manager who cares about my development and helps me identify and achieve my goals. If those aren't happening? I'm probably going to have a wandering eye, looking for better opportunities.

It's long been a quest of mine to figure out what a great manager looks like. I don't know if I want to be a manager yet, but I have pretty strong feelings about how I want people who manage me to behave.

While on that quest, I recently (re-)discovered a 2013 report by HBR on how Google researched and analyzed management to make Google a place top notch engineers want to work. I think the key takeaways are their top 8 behaviors of good managers:

A good manager:
1. Is a good coach
2. Empowers the team and does not micromanage
3. Expresses interest in and concern for team members’ success and personal well-being
4. Is productive and results-oriented
5. Is a good communicator—listens and shares information
6. Helps with career development
7. Has a clear vision and strategy for the team
8. Has key technical skills that help him or her advise the team

The other key takeaway is that in order to make sure managers knew what they needed to improve on they instituted a regular "Upward Feedback Survey" that would allow reports to provide numeric feedback about their managers in those 8 key areas. That feedback was then aggregated and reported back to each manager, so that they could improve.

Plus, this feedback loop was outside of the normal performance reviews. It was confidential and allowed managers to improve themselves. It's a development tool, not a performance metric.

Then on top of that Google provided optional management training courses, for managers to improve in the areas their feedback suggested.

It's really that simple.

1. Create a feedback loop
2. Analyze and report the metrics
3. Allow people to improve on personal areas of weakness without threat to their job

People want to improve. If they don't, you hired the wrong people.

TDD Is a Tool, Not a Religion

Catching Up

The internet is abuzz since DHH's blog post: TDD is Dead.
There have been many responses, including many by Bob Martin.

Breaking News

The latest new entry to the saga is a video conversation on YouTube where DHH, Martin Fowler and Kent Beck discuss the virtues and pains of test driven development. Definitely worth watching.

Subjective Summary

1. TDD scratches an itch. It's one way to make you feel good about your work, confident that it works as intended, every step of the way. 
2. TDD does not work well for all problems, like when the output is unknown or unquantifiable.
3. Self-testing code (programmatic & repeatable) is highly desirable, but TDD is not the only way to get there.
4. Heavy use of reflective mocking can tightly couple tests to implementations, counteracting one of the primary reasons to have tests: to enable refactoring. (Bob Martin has some thoughts on when and how to mock.)
5. Programming has many trade offs (testability, maintainability, usability, speed, etc.), and it is important to keep them all in perspective, re-prioritizing as needed, without becoming myopic.

Corollary

Just because writing unit tests and making them pass is easier and provides faster feedback than end-to-end tests doesn't mean unit testing is more important. It's critical that good unit design doesn't come at the cost of good system design.

I'll leave you with my favorite list of Object Oriented Design Principles. As with most sets of goals, it's impossible to satisfy them all all the time, but it's important to keep them all in mind, and know when to apply them.

Capitalism + Socialism = America 2.0

Anthony Bourdain wrote a nice perspective piece on gun culture in America. It made me think.

Guns may not be a part of your culture, they're not part of mine, but they're definitely part of America's culture, past present and future. They are definitely deadly weapons, but so are cars. Neither are going away anytime soon.

Guns may not be what congress is fighting over today, but different cultures are definitely clashing. Do we individually pull ourselves up by our own bootstraps or do we collectively pull each other up? Do you spend your hard earned money to help some stranger you don't even know? Are people in need fellow humans down on their luck or mooching free loaders? That's what the fight is over.

If you're the mathematical type you might even recognize both issues from game theory, the study of conflict and cooperation between intelligent rational decision-makers. We could jest about congressional intelligence and rationality, but lets not miss the wide ranging implications. Eight game-theorists have won the Nobel prize, including John Nash for his idea of Nash equilibrium, the idea that the optimal solution for multiple competing game players is not just to play your own strategy but to adapt your strategy to the strategies of your competitors. If you ignore everyone else's priorities and goals and strategies you will lose.

I've applied this to corporate politics already, but it also applies to national politics. Granted, there are many complexities, like Bayes-Nash equilibrium, differing belief systems, random chance, incomplete information, risk avoidance or corruption. The core concept is still intuitive enough to apply (obviously the details are non-trivial). In general tho, we can't just think of ourselves and we can't just think of the collective. We need to think about what's best for us AND our community, me AND my neighbor, the state AND the country.

Our constitution may be a great document, but it's woefully lacking in these kinds of mixed-strategy incentives. It relies primarily on checks and balances, a strategy for combating the aggregation of power, but checks and balances are mostly disincentives, not incentives to do the right thing. Our constitution is great at stopping any one person or branch of government from becoming too powerful, but it's pretty terrible at motivating optimal behavior or continual improvement. What feedback loop is there to motivate your congressional representative to think about what's best for the country? If they please their constituents they get elected again, but where's the motivation to come to the table and do what's best for the other 99%+ of the people in the country? Instead of multiple incentives and enlightened thinking we have a government run on head butting competition that enables and glorifies pure selfish greed.

"Screw you, got mine" selfishness is poison the same way utopianaltruism is. Cooperation is nothing without a little self-interest, but too much self-interest unbalances the system so that everyone loses. Romanticized Capitalism is just as flawed as utopian Socialism. It's time for a new -ism; one where we think about us AND them.

Exponential Improvement Is The New Growth

Here's a sneak peak at a future S.A.T. analogy. Fill in the blank.

Velocity : Acceleration : Jerk :: Growth : Continuous Improvement : _________

We'll get to the answer, but first a history lesson!

Buckminster liked to make up new words. "Ephemeralization" was his from the 1930's. The classic definition of ephemeralization is "doing more with less", but it's more precisely the observation that things happen more quickly and easily over time, requiring less resources. The easier things become, the faster they become easier. The cheaper things become, the faster they become cheaper. Generally this is attributed to the adoption of scientific management, as exemplified in early industrialization (especially by Henry Ford).

Gerald Hawkins in the 1980's described the idea that each stage of change had an inflection point where a paradigm shift ('mindsteps', as he called them) occurred, advancing the rate of change forward. He also observed that the rate of paradigm shifts was increasing. Change was accelerating.

In 1986, Masaaki Imai popularized "Kaizen", the idea of continuous improvement in business management. Ideally improvement doesn't need to be limited to discrete (continual)  jumps based on scheduled analysis after designated periods of time (iterations), but could instead be driven by slack in the system to allow (continuous) improvements every day. The acceleration of change requires faster measurement and analysis, but with built-in slack the workers on the factory floor can do it better than managers at a distance.

In 2010, David Anderson documented "Kanban", a method of software development that integrates Agile concepts with Japanese Kaizen improvement techniques. The practices and policies Anderson describes are great management techniques based on years of experience and an evolution of Agile methodology. His primary focus is on work in progress limits to create slack, but the real power comes from empowering workers with information and constant feedback so that they can use the slack for continuous improvement.

So with the implementation of continuous improvement in Japanese factories (like Toyota) and later Silicon Valley software developers (like Google), managers found themselves with less management to do because they'd empowered their employees to do their job for them. Many people have been confused by this, thinking that managers are now obsolete in an organization of continuous improvement (like Valve software), but the truth is simply that managers need to level up, the same way factory workers replaced by automation had to become educated to produce, maintain, and improve automation. Managers won't manage people in the future, they'll manage their own empowerment of people.

Managers don't just provide feedback any more. They empower employees to provide and receive feedback. Just like employees need feedback to improve, management needs feedback to improve as well. If you really want to get better at getting better you need to level up again. "It's much less about 'What kind of knowledge advantage do I have right now?' but 'How fast am I creating new knowledge?'."

This is where executives come in. The new job of executives isn't to manage the managers any more or even just to manage external expectations, but to empower the managers to manage themselves. Empowered managers get better at empowering workers to get better, inovating on innovation. As we've seen on the previous level, old style executives are becoming obsolete, even a liability. They need to educate themselves and raise to the challenge of faster change, to manage the empowerment of managers. They need to not just set a culture, but grow a culture of change. Business doesn't just need to evolve; it needs to improve its ability to evolve.

Back to the analogy...

Velocity is position over time. Acceleration is velocity over time. Jerk is acceleration over time. Growth is improvement of state over time (the new role of workers). Continuous Improvement is positive growth over time (the new role of managers). So Exponential Improvement is faster continuous improvement over time (the new role of executives).  Jerk is the third time-derivative of position. Exponential Change is the third time-derivative of state. Exponential Improvement is positive Exponential Change.

No, this doesn't mean future executives will all be jerks, but they might need to be good at calculus. The executive needs to enable Exponential Improvement, and the easiest way to do that is to derive it from the employee improvement metrics over time. They'll have to analyze metrics and behavior to figure out how to improve exponentially. They'll have to identify synergistic ways to empower managers that increase their rate of empowerment.

To be fair, Continuous Improvement will get you a long way. Most organizations haven't even figured out that yet, but they will. And the more that figure it out, the faster the movement will grow. But if you really want to succeed in the future you can't stop there. You have to get better at getting better at getting better.

Future Shock is no longer in the future, it's now. Change is and will be. Change is continuous and accelerating and decentralized. This isn't a revolution; revolutions end. Businesses that didn't change are already dead. Businesses that aren't changing fast enough are dying. The new race is how fast you can innovate. It's a race you can't conclusively win; you can only win now and survive until you win again. Growth is slowing down. To grow linearly you'll need to improve exponentially.

Stop watching the world improve and start participating in making it improve faster.

SteamLevel

SteamLevel.com Forwards Here.

SteamLevel was a site for comparing gamer profiles on Valve's Steam digital distribution platform.

The primary reason I wrote SteamLevel was to be able to decide what game to play at LAN parties with my friends. We could add all the people there to the search list and it would generate a big table of all our games, making it easy to see which ones we all had.

But hosting a web application is expensive. There was no ad revenue and not many users. I tried Amazon sponsered links to games, but that just felt like a slap in the face to my users, who were there because of Steam, not Amazon's digital distribution platform.

But the real problem was that I had heavily cached everything. So it took gobs of memory. And web hosts really hate to give you memory. Memory turns out to be the hardest thing to increase. It's easier to get a dozen CPU cores than it is to get more than 4GB of memory, and my host had capped me at 2GB. I was hosed. The application needed a rewrite to even continue using that web host and anything with more memory was 10x as expensive. I just really didn't want to pay that much for hosting. it would have been cheaper to buy my own box and upgrade my ISP to business class.

So SteamLevel is down, indefinitely.

It may get resurrected at some point. I still own the domain. But not today. Sorry.

Tablets: 7" vs 10"

Seven inch tablets are better at two thumbed typing in portrait orientation, while ten inch tablets are better at two handed typing in landscape on a flat surface.

Seven inch tablets are small enough to fit in purses and large pockets.

Seven inch tablets are holdable in one hand for extended periods.

If you want to use your tablet like a computer, get a 10 inch. If you want to use it more like a smartphone, get a 7 inch.

Listening to music on the go on any tablet makes you look silly. Thankfully people don't care enough to comment.

I've based the above generalizations solely on my own iPad 3 and Nexus 7.

Updating Nexus 7 (2013) from JWR66N to JSS15Q on Windows

So the Nexus 7 had 2 issues at launch: GPS & Multitouch. 

The launch android build that comes with the Nexus 7 is JWR66N. The fixes are in JSS15Q.
Find your build number: Settings > About Tablet > Build Number

There was a patch soon after launch, JSS15J, but it doesn't fix either of the above issues. If you don't already have JSS15J google probably wont give it to you over the air (OTA) any more, because it doesn't fix the issues and might make them worse. If you already have JSS15J don't worry, you can still upgrade.

Now if you're squeamish about developer mode you shouldn't be here, just wait for Google to give you JSS15Q over the air. But if you want to get your hands dirty you can resolve those pesky issues now!

1) You'll need developer mode enabled on your tablet.
http://www.androidcentral.com/how-enable-developer-settings-android-42

2) You'll need to change the USB mode on your tablet to "Camera(PTP)".
http://zacktutorials.blogspot.ca/2012/08/nexus7-android-development.html

3) You'll want "stay awake" and "usb debugging" enabled in your developer options on your tablet.
http://zacktutorials.blogspot.ca/2012/08/nexus7-android-development.html

4) You'll need to download and unzip the Android SDK for Windows.
http://developer.android.com/sdk/index.html

5) Launch the "SDK Manager" in the zip you just downloaded and install the "Google USB Driver". (It'll probably ask to install the SDK and Android 4.3 while you're at it.)

6) Modify the USB driver to add the device ID of the nexus 7 when in recovery mode.
http://blog.dantup.com/2012/10/fixing-adb-device-not-found-with-nexus-7-in-recovery-mode

7) Plug in your tablet via USB (it will attempt to install drivers if it's the first time or it has failed before).

8) Install the newly modified device driver.
- Go to the Windows Device Manager, select your tablet from the "Other devices" category, right click and select "Update Driver Software".
- Opt to browse your computer for driver software and point it to "\extras\google\usb_driver".
- Note that the modified driver won't pass checksum and thus wont pass signature validation. So you'll have to approve it when it warns you.

9) Run adb to request device debug authorization.
- Open the command prompt (cmd in the start menu).
- Navigate to \platform-tools
- Run "adb devices"
This should print out one device as unauthorized.

10) On your tablet, authorize your computer. Select "always allow from this computer".

11) Run "adb devices" again to make sure your device says "device" and not "unauthorized".

12) Download the "JWR66N->JSS15Q OTA" or "JSS15J->JSS15Q OTA" depending on which version you have on your tablet (Settings > About Tablet > Build Number).
http://forum.xda-developers.com/showthread.php?t=2415497

13) Use adb to reboot your tablet into recovery mode and side load the update.
- Follow Step 11 onward from http://www.droid-life.com/2013/02/12/guide-how-to-use-adb-sideload-to-update-a-nexus-without-root-or-custom-recovery/

14) Do a happy dance!

For the record, I found the responses on this stack overflow question helpful when trying to figure this all out:
http://stackoverflow.com/questions/11974700/nexus-7-not-visible-over-usb-via-adb-devices-from-windows-7-x64#14106931

Apparently this process is easier on the Mac, because the device drivers don't have to whitelist device IDs and "just work".

Iterable is an Anti-Pattern

I couldn't decide I hated UnsupportedOperationException or Iterator more, so I decided to pick on Iterable instead...

Iterable limits the implementer to having one interesting iterable attribute. 

Having a single iterator producing method (Iterable.iterator()) restricts the implementing object to only ever have one interesting quality to iterate over. This requires that it be obvious from the name of the iterable class and the class name of the iterable components (that you often can't control) what it is you're allowed to iterate over.

Company workCo;
for(Position currP : workCo) { ... }

What are we iterating here? Positions within the Company? What does that even mean? Is it even ordered? Is it open positions or filled positions? These are all things you'd normally hit at with a well named method. 

The JDK authors at least sort of figure this out for Map, but in order to make it work they had to expose the inner workings of how a Map works, and in doing so dictate implementation details in an interface. Your Map must implement entries() and it can't just be a view; it has to be mutable (or explosive) and tightly coupled with the Map it came from. This is terrible for asynchronous execution and expensive to implement.

Now, to play devil's advocate, one might say that if you have multiple iterable attributes you shouldn't be iterable... That's valid. Maybe the Company class really needs to contain multiple iterable collections, like currentEmployees() and openPositions(). But this just hammers home my point by pulling "an Apple" and telling users they're holding/using it wrong.

Iterable is not extensible.

Your object may only have one iterable element, but what about extensions? Iterable forces any extensions with any other iterable attributes to be via compositions and views. This is especially true for elements with multiple attributes or multiple views.

Liskov's Substitution Principle requires that all extensions demonstrate an "is-a" relationship to their parent. If you missed that day in CS class, it means that "Nissan370Z" can extend "Car", but "Truck" can't. Even if a trunk does all the same things a car does, a truck is not a car.

The more API you define the harder it is to extend. 

Iterable is poorly composable. 

So when you can't extend to add another view type what do you do? You compose the class with a view.

class Sizes implements Iterable {
  Collection> stuff;
  Viewer(Collection stuff) {
    this.stuff = stuff;
  }
  Iterable iterable() {
    return new Iterator() {
      [impl goes here]
    };
  }
}

So you took your collection of collections and composed it, and now you get to write your own custom Iterator... Yay.

Terrible. 

Interface method names shouldn't hide what they return. 

If an object has multiple Iterable fields, for example, and it implements Iterable... which field does the returned Iterator, from the method iterator(), actually iterate over? The method name doesn't tell you, and that means the code that iterates the object doesn't tell you, and is thus less readable.

Abstract the type, sure, but don't abstract the method name. Otherwise you end up with and API with:

Iterator iterator();

What does that return?
It might make sense if the type is more specific, but are you really going to wrap your String in an object just to give it a better name? No, that's stupid and expensive.

You shouldn't have to implement something to say you're not doing it.

Iterator is an anti-pattern itself, with its mutability baked right in. You have to explode with an UnsupportedOperationException if you want to tell the user (at runtime) that the view you're iterating over isn't mutable.

@Override
public void remove() {
    throw new UnsupportedOperationException("Sorry, No.");
}

How many times have you had to write that crap?
Google Guava even has an abstract class to clean up their code, but that doesn't legitimize it.

To be honest, UnsupportedOperationException is pretty bad all by itself. Just it's existence means that the JDK regularly violates the Liskov's Substitution Principle.

All JDK collections are mutable. 

This is really one of the core problems with the JDK and the reason Iterator is bad. The API for immutable objects is conceptually a subset of a mutable API with the primary exception that a builder or constructor from a mutable is required. 

So what's the fix?

There isn't one really. The JDK is hosed. It's never going to get fixed, because it would break reverse compatibility horribly, and because we can't delete those old classes we can't really reuse their names either, they're already taken.

I've considered spinning up an alternative JDK, starting with collections, taking what we've learned about immutability and substitution and composition and starting over.... Who's with me?

We'll start with this:

interface ForwardCursor {
  boolean hasNext();
  T next();
}
interface BackwardCursor {
  boolean hasPrevious();
  T previous();
}
interface BidirectionalCursor extends ForwardCursor, BackwardCursor {
}

And we'll add some magic syntactical sugar similar to the for each loop to work with ForwardCursor, except we're goign to call it "for rest" because it will start where the cursor is currently, and never needs to worry about making a new cursor. That way your object can have multiple cursor'd attributes, each with their own methods:

CollectionOfCollections {
  Collection> collections();
  BidirectionalCursor sizes();
}

Granted, that last usage example is a little silly... Let me know if you come up with a better one!

Technology in Education

This article (linked in the title) from the NY Times reports on the selective happenings of some students and how they are affected by the general trend towards increased computer use and constant connectivity. The author tries to stay objective in the reporting, but the picture painted is one of change and the difficulties educators have adapting to that change.

While technology may be difficult to adapt to it is not a fad. One of the educators quoted seems to compare advancing tech to rock and roll. Not only does this date the speaker but it also indicates a misunderstanding of technology in general. Rock and roll was and continues to be a category of a narrow section of consumable media. At most one could classify it as a lifestyle. But technology is a much wider empowerment of all lifestyles (save those that reject it).

It's plainly obvious that educators are seeking ways to use technology to better reach students. While this is a worthwhile goal, perhaps it might be more useful to invest in improving motivation, curiosity, and useful curiculum.

As seen in the article, many students are motivated by trade skills like video and audio editing, useful skills that students can easily identify as applicable to their future. I'm not implying that all schools should be trade schools, but the world that values renaissance men and women has past. As technology advances the volume of availible skills and information increases dramatically. Many large corporations spend millions categorizing information and educating their work force. As information explodes it becomes increasingly difficult for one person to know enough about general topics to be any more knowledgable than a search engine. Knowing facts becomes less and less important. Facts can be categorized and searched. Yet schools continue to rely heavily on memorizing things that can be easily looked up on any student's wireless device. It's no wonder students don't like learning algebra or Latin. The modern world simply doesn't need EVERY student to be able to do these things.

At university they used to like to tell us they were teaching us how to learn. Why did it have to wait until university? Why wouldn't I know how to learn before that? We need to teach kids how to learn much earlier. Then we can concentrate on teach skills, not just knowledge.

More Cool jQuery Plugins

There's so many to choose from, but no easy way to find what you need.
Here's some more examples I like.

Cufonized Fly-out Menu: Vertical menu with sliding hover highlight and supplemental fly out descriptions.

Star Rating: Rate something with (5) stars and half stars.

Lazy jQuery Plugin Loader: Load jQuery js files on demand

Inline Form Validation: Tooltips for confirmation or error messages

Auto Suggest: Auto-completion similar to many email clients

FullCalendar: A calendar similar to Google's that you can populate with events or even use with Google's data.

Cool jQuery Plugins

I thought it was time I highlighted a smattering of useful and cool jQuery plugins. While there are many of these lists out there it's always nice to have new lists pop up to bring up the popularity of the best plugins. It can be hard to browse the master list and sometimes you dont see the coolest ones!

First and foremost I want to point out jQuery UI. If you aren't using it you're doing it wrong! It's a great place to start when looking for common javascript effects and widgets and themes.

And now in no particular order...

Labelify: Put text inside your text input boxes that disappears when you select the input.

Cycle Plugin: Transition effects for your divs (think Power Point slides).

ColorBox: UI dialog widget for modal windows. Check this out if jQuery UI's modal windows don't suit your needs. Great for image viewing.

Login Form Tutorial: Hidden form that slides in/out.

Table Sorter: Turn a normal table into sortable one!

Uniform: Themed form elements. Looks the same in all browsers.



50 (More) Amazing jQuery Examples

Pet Peaves = Annoyance Revenge

It may bother you, but as soon as you complain you are now bothering everyone else with it too. Congratulations! You are now more annoying than the thing that annoys you and you probably just exacerbated the problem.

CarFM by pdo Review

NOTE: This review is for the first revision CarFM. The second revision seems to have removed the audio wire and added portrait rotation.

I don't usually do reviews and this probably isn't the best place to post a review that people will actually be able to find, but here goes.

I decided it was time to buy a charger, stand and fm transmitter for my iPhone. Previously I had been using the ubiquitous Griffin iTrip FM Transmitter and Car Charger with my 3rd generation iPod, but the old version I had didn't work with my iPhone so I basically decided to leave the iPod in the car. However, I recent decided I needed a stand for my iPhone to better use the maps and GPS while driving. Also, I don't have an auxiliary input on my car stereo. I'd prefer that over an FM transmitter any day.

So I researched a few of the FM Transmitters and stands that work with the iPhone and found none of them perfect, and most of them expensive. I finally settled on a CarFM by pdo because the price was right ($25-$45 depending where you buy) and the reviews were decent.

This is what I've discovered:

1) The CarFM has a headphone wire as well as an apple jack. This allows you to control the transmitter's input volume, but otherwise isn't very useful and requires plugging in two things instead of one. Also your volume will probably be maxed for the car, which means when you later use your iPhone with headphones the volume will be at max, beware!

2) The FM transmitter has a superior signal to any I've used before.
This has partially to do with the fact that it's connected by metal to your dashboard and partially just because it has a strong signal. However, it is so strong that nearby stations gain static. For instance set to 106.1 the transmitter makes the normally clear KROQ on 106.7 unlistenable.

3) The transmitter is always transmitting when plugged into the cigarette lighter, even if no iPhone/iPod is plugged in. I wish there were an off button to save my car battery a bit of charge. I think it's also still transmitting when the car is off, at least all the lights stay on, at least until the car cuts power to the cigarette lighter.

4) Receiving a phone call stops the music and puts the phone on speaker (iPhone speaker, not car speakers). This is essentially useless in a running car. I haven't tried it with a bluetooth headset yet, that might work better.

5) The stand seems to be sturdy enough in my car, better than I thought it would be, but it's still only held up by a circular connector, which might slip more over time.

6) The stand's flexible wire is obviously fixed length. In my case the length is pretty good, better than the short competitors.

7) The auto-seek button works pretty well at finding an empty station, but the transmitter is pretty strong so it doesn't have to be perfect anyway.

8) The station is on the top of the display which is a little awkward, but really if the signal is good you don't really need to look at after the initial setting. I'd recommend making a radio bookmark for the station you pick.

9) The station light and blue indicator light on the charger are always lit, which may annoy you and sucks a tiny amount of power. Not a big problem compared to the transmitter always transmitting.

If that sounds sufficient to you then go for it. It does what it says it does, but it's not perfect.

Lua Deep Equals

I was surprised when I couldn't easily google for a Lua deep compare function. It's main purpose is to compare tables and check if they have the same values, but of course it's recursive so it just uses ~= to check for difference in non-tables.

Anyway, so I wrote my own and figured I'd post it here for posterity. Maybe you can come up with a shorter version?

-- Deep Equals
local function equals(t1, t2)
   if t1 == t2 then
       return true
   end
   if type(t1) ~= "table" or type(t2) ~= "table" then
       return false
   end
   local v2
   for k,v1 in pairs(t1) do
       v2 = t2[k]
       if v1 ~= v2 and not equals(v1, t2[k]) then
           return false
       end
   end
   for k in pairs(t2) do
       if t1[k] == nil then
           return false
       end
   end
   return true
end

Godiva Float

I just died... and went to heaven... and I've come back to give you the recipe...

Ingredients:
-Godiva Chocolate Liqueur
-Bailey's Irish Cream
-Ben & Jerry's Chocolate Fudge Brownie Ice Cream

Instructions:
1) Put them all together in a mug.
2) ...
3) Profit!

Oh man, you have no idea. You NEED to try this if you consider yourself a chocoholic. Otherwise.... no... there is no otherwise! You NEED to try this!

Favorite tsch Aliases

So I spend a lot of my time at work tooling around on a remote server through ssh. Here are a few of the aliases I use everyday that make my life a little simpler. Put them in your login script and save yourself some headaches!

#setup color ls
alias ls ls-F ; set color=ls-F

#ls shortcuts (cause 2 chars is better than 4!)
alias la 'ls -a'
alias ll 'ls -lg'

#vim color shortcut
alias vi 'vim -X'

#make grep ignore svn files
alias grep 'grep \!* | grep -v /\.svn/'

#remove those pesky Permission denied lines when using find
alias find 'find \!* |& grep -v /Permission denied/'

#change dir and list contents
alias cdls 'cd \!*; ls'