Month: January 2015

Linux and Solaris are Converging but Not the Way You Imagined

tux

In case you haven’t been paying attention, Linux is in a mad dash to copy everything that made Solaris 10 amazing when it launched in 2005. Everyone has recognized the power of Zones, ZFS and DTrace but licensing issues and the sheer effort required to implement the technologies has made it a long process.

ZFS

ZFS is, most probably, the most advanced file system in the world. The creators of ZFS realized, before anyone else, that file systems weren’t built to handle the amounts of data that the future would bring.

Work to port ZFS to Linux began in 2008 and a stable port of ZFS from Illumos was announced in 2013. That said, even 2 years later, the latest release still hasn’t reached feature parity with ZFS on Illumos. With developers preferring to develop OpenZFS on Illumos and the licensing issues preventing OpenZFS from being distributed as part of the Linux Kernel, it seems like ZFS on Linux (ZOL) may be doomed to playing second fiddle.

DTrace

DTrace is the most advanced tool in the world for debugging and monitoring live systems. Originally designed to help troubleshoot performance and other bugs in a live Solaris kernel, it quickly became extremely useful in debugging userland programs and run times.

Oracle has been porting DTrace since at least 2011 and while they both own the original and have prioritized the most widely used features, they still haven’t caught up to the original.

Zones

Solaris Zones are Operating System level virtual machines. They are completely isolated from each other but all running on the same kernel so there is only one operating system in memory. Zones have great integration with ZFS, DTrace, and all the standard system monitoring tools which makes it very easy to support and manage servers with hundreds of Zones running on them. Zones also natively support a mechanism called branding which allows the kernel to provide different interfaces to the guest zone. In Oracle Solaris, this is used to support running zones from older versions of Solaris on a machine running a newer OS.

Linux containers of some type or another have been around for a while, but haven’t gotten nearly as mature as Zones. Recently, the continued failure of traditional hypervisors to provide bare metal performance in the cloud, coupled with the uptake of Docker, has finally gotten the world to realize the tremendous benefits of container based virtualization like Zones.

The current state of containers in Linux is extremely fractured with at least 5 competing projects that I know of. LXC, initially released in 2008, seems to be the favorite but historically had serious privilege separation issues but has gotten a little better if you can meet all the system requirements.

Joyent has been waiting at the finish line.

While Linux users wait and wait for mature container solutions, full OS and application visibility, and a reliable and high performance file system, Joyent has been waiting to make things a whole lot easier.

About a year ago, David Mackay showed some interest in Linux Branded Zones, work which had been abandoned in Illumos. In the spring of 2014, Joyent started work on resurrecting lx-zones and in September, they presented their work. They already have working support for 32 bit and some 64 bit Linux binaries in Linux branded SmartOS Zones. As part of the process, they are porting some of the main Linux libraries and facilities to native SmartOS which will make porting Linux code to SmartOS much easier.

The upshot of it is that you can already get ZFS, Dtrace, and Linux apps inside a fully isolated, high performance, SmartOS zone. With only 9 months or so of work behind it, there are still some missing pieces to the Linux support, but, considering how long Linux has been waiting, I’m pretty sure SmartOS will reach feature parity with Linux a lot faster than Linux will reach feature parity with SmartOS.

Your Graphic Designer is Tanking Your Site!

mona lisa

Your graphic designer is an artist, a trained expert in aesthetics, a master at conveying messages via images and feelings via fonts. He may also be slowing your site down so much that nobody is seeing it.

Artists tend to be heavy on the quality and lighter on the practicality of what they deliver. It’s not entirely their fault. Even the most conscientious and experienced designer needs to sell his work and quality sells. Do marketing departments want to see their company advertised in 4K glory or on their mom’s 19″ LCD?

Quality isn’t worth the cost.

The reality of the Internet is that too much quality costs more than it’s worth. It costs bandwidth and it costs customers who aren’t willing to wait for heavy sites to load.

Is there a subliminal value to high quality graphics? The answer is yes but only if someone sees them.

How much do the images make a difference?

Here is a quick experiment you can do to visualize some of the improvement you could see. Go into your browser settings and disable all images. Then go back and visit your website to see how it loads without the bloat.

I just tried this on a the homepage of a major news network. With images the site was over 4MB. You don’t even see most of the pictures that were loaded. Without images the site was under 2MB (still very high to be honest). That basically means that, according to the laws of physics and in the best case scenario, the site with images will take at least twice as long to load.

You might say they are a huge media site. They know what they’re doing. They need the images. The sad truth is that they are wasting your bandwidth for no good reason as I’ll demonstrate shortly.

How to fix it?

I won’t tell you to get rid of all the images on your site, though if that can fit with your needs, less is always faster. You do, however, need to optimize your images for the web.

As a performance engineer who has examined the performance of literally hundreds of websites, the number one problem is always images that haven’t been optimized. Just optimizing your images properly can cut the download time of a website in half, possibly more.

As a continuation of our experiment above I took this picture from their page and tried optimizing it to see what kind of savings I could get. Here is the original 41KB version:

Here is the optimized 15KB version:

The result is almost 1/3 the size and I don’t think you can tell the difference. Would you notice any difference if this was just another image on a website? You too could get a 50-60% performance boost, just by optimizing your images.

If you are using some type of automated deployment for your sites, there are tools which will optimize your images automatically. They are OK for basic optimizations.

To really optimize your images, you need a person with a realistic eye (graphic designers are always biased towards heavier higher quality images) and you need to follow these basic rules:

Use the right images for the job.

In general, properly compressed, Progressive JPEG files are the best to use but they don’t support transparency. After weighing carefully if you need a transparent image or not, use Progressive JPEG files for any image that has more colors than you can count on your hand and doesn’t require transparency. Otherwise use a PNG file.

Optimize the images.

Optimizing JPEG files

First, manually reduce the file’s quality level until you reach the setting with acceptable levels of pixelation in the sharp edges and gradient surfaces of the image.

Note that you should always start optimizing from the original image at it’s best quality. Repeatedly editing a JPEG file will degrade the quality of the image with each generation.

After you have reached the optimal quality level, run them through a tool likeImageOptim which will remove any thumbnails or metadata adding weight to the image.

Optimizing PNG files

Optimize PNG files first by running them through a tool like ImageAlpha orTinyPNG. These tools use lossy compression techniques to reduce the number of colors in your image fitting them into a smaller bitmap, and resulting in better compression than a PNG would normally have.

Note: ImageAlpha gives you more control over the process letting you decide how many colors should be used in the resulting image. This is very useful for transparent PNGs with very few colors.

Then run the images through ImageOptim or similar tool to squeeze some extra space out of them.

Once you have mastered the above, your site should be much lighter. If you want to optimize further (and you should), look into the following techniques:

Don’t use images where you don’t have to.

Many of the most common icons used on websites have been implemented as fonts. Once a font is loaded, each icon is a single character and making the icon larger or smaller is as simple as using a larger font size.

Note: There is some overhead in loading and using the fonts (additional CSS) so use wisely.

Combine small images.

Similar to the idea of using fonts, is the CSS Sprite. These combine multiple small images into a single transparent PNG file and then show only one image at a time on your site using CSS tricks.

The advantage of this technique is that it saves requests to the web server. Each request to a web server has latency, sometimes as much as 200ms, which can’t be eliminated and for small images, this request latency can sometimes account for more time than downloading the image itself. By combining the smaller images in your site, you save that overhead.

There are tools which will generate the combined images and the corresponding CSS for you.

Summary

I’ve used these image optimization techniques in hundreds of websites resulting in significant savings of bandwidth and increased performance. Most importantly, though, these techniques result in better visitor engagement.

If you’re interested in optimizing your site for best cost/performance, feel free to contact me via LinkedIn or via https://donatemyfee.org/.

SmartDataCenter, the Open Cloud Platform that Actually Already Works

sdc

For years enterprises have tried to make OpenStack work and failed miserably. Considering how many heads have broken against OpenStack, maybe they should have called it OpenBrick.

Before I dive into the details, I’ll cut to the chase. You don’t have to break your heads on cloud anymore. Joyent have open sourced (as in get it on Github) their cloud management platform.

It’s free if you want (install it on your laptop, install it on a server). It’s supported if you want. Best of all, it actually works outside of a lab or CI test suite. It’s what Joyent runs in production for all their public cloud customers (I admit to being one of the satisfied ones). It’s also something they have been licensing out to other cloud providers for years.

Now for the deep dive.

What’s wrong with OpenStack?

First off, it isn’t a cloud in a box, which is what most people think it is. In 2013,Gartner called out OpenStack for consciously misrepresenting what OpenStack actually provides:

no one in three years stood up to clarify what OpenStack can and cannot do for an enterprise.

In case you’re wondering, the analyst also quoted Ebay’s chief engineer on the true nature of OpenStack:

… an instance of an OpenStack installation does not make a cloud. As an operator you will be dealing with many additional activities not all of which users see. These include infra onboarding, bootstrapping, remediation, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasting and management, billing or chargeback, reclamation, security, firewalls, DNS, integration with other internal infrastructure and tools, and on and on and on. These activities are bound to consume a significant amount of time and effort. OpenStack gives some very key ingredients to build a cloud, but it is not cloud in a box.

The analyst made it clear that:

vendors get this difference, trust me.

Other insiders put the situation into similar terms:

OpenStack has some success stories, but dead projects tell no tales. I have seen no less than 100 Million USD spent on bad OpenStack implementations that will return little or have net negative value.Some of that has to be put on the ignorance and arrogance of some of the organizations spending that money, but OpenStack’s core competency, above all else, has been marketing and if not culpable, OpenStack has at least been complicit.

The motive behind the deception is clear. OpenStack is like giving someone a free Ferrari, pink slip and all but keeping the keys. You get pieces of a cloud but no way to run it. Once you have put all your effort into installing OpenStack and you realize what’s missing, you are welcome to turn to any one of the vendors backing OpenStack for one of their packaged cloud platforms.

OpenStack is a foot in the door. It’s a classic bait and switch but even after years, no one is admitting it. Instead, blue chip companies fight to steer OpenStack into the direction that suits them and their corporate offerings.

What’s great about SmartDataCenter?

It works.

The keys are in the ignition. You should probably stop reading this article and install it already. You are likely to get a promotion for listening to me 😉

Great Technology

SmartDataCenter was built on really great technologies like SmartOS (fork of Solaris), Zones, ZFS, and DTrace. Most of these technologies are slowly being ported to Linux but they are already 10 years mature in SDC.

  • Being based on a fork of Solaris brings you baked in enterprise ready features like IPSEC, IPF, RBAC, SMF, Resource management and capping, System auditing, Filesystem monitoring, etc.
  • Zones are the big daddy of container technology guaranteeing you the best on-metal performance for your cloud instances. If you are running a native SmartOS guest, you get the added benefit of CPU bursting and live machine re-size (no reboot, or machine pause necessary).
  • ZFS is the most reliable, high performance, file system in the world and is constantly improving.
  • DTrace is the secret to low level visibility with zero to no overhead. In cloud deployments where visibility is usually close to zero, this is an amazing feature. It’s even more amazing as the cloud operator.

Focus

SDC was built for one thing by one company, to replace the data centers of the past. It says so in the name. With one purpose, SDC has been built to be veryopinionated about what it does and how it does it. This gives SDC a tremendous amount of focus, something sorely lacking from would-be competition like OpenStack.

Lastly, it works.

Couchbase is Simply Awesome

couchbase

Here are five things that make Couchbase a go-to service in any architecture.

Couchbase is simple to setup.

Keep It Simple. It’s one of the axioms of system administration. Couchbase, though complicated under the hood, makes it very simple to setup even complicated clusters spanning multiple data centers.

Every node comes with a very user friendly web interface including the ability to monitor performance across all the nodes in the same machine’s cluster.

Adding nodes to a cluster is as simple as plugging in the address of the new node after which, all the data in the cluster is automatically rebalanced between the nodes. The same is true when removing nodes.

Couchbase is built to never require downtime which makes it a pleasure to work with.

If you are into automation a la chef, etc., Couchbase supports configuration via REST api. There are cookbooks available. I’m not sure about other configuration management tools but they probably have the relevant code bits as well.

Couchbase replaces Memcached

Even if you have no need for a more advanced NoSQL solution, there is a good chance you are using Memcached, Couchbase is the original Memcached on steroids.

Unlike traditional Memcached, Couchbase supports clustering, replication, and persistence of data. Using the Moxi Memcached proxy that comes with Couchbase, your apps can talk Memcached protocol to a cluster of Couchbase servers and get the benefits of automatic sharding and failover. If you want, Couchbase can also persist the Memcached data to disk turning your Memcached into a persistent, highly available key value store.

Couchbase is also a schema-less NoSQL DB

Aside from support for simple Memcached key/value storage, Couchbase is a highly available, easy to scale, JSON based DB with auto-sharding and built in map reduce.

Traditionally, Couchbase uses a system called views to perform complicated queries on the JSON data but they are also working on a new query language called N1QL which brings tremendous additional ad hoc query capabilities.

Couchbase also supports connectivity to Elastic Search, Hadoop, and Talend.

Couchbase is all about global scale out

Adding and removing nodes is simple and every node in a Couchbase cluster is read and write capable all the time. If you need more performance, you just add more nodes.

When one data center isn’t enough, Couchbase has a feature called cross data center replication (XDCR), letting you easily setup unidirectional or bidirectional replication between multiple Couchbase clusters over WAN. You can even setup full mesh replication though it isn’t clearly described in their documentation.

Unlike MongoDB, which can only have one master, Couchbase using XDCR allows apps in any data center to write to their local Couchbase cluster and that data will be replicated to all the other data centers.

I recently setup a system using five Couchbase clusters across the US and Europe, all connected in a full mesh with each other. In my experience, data written in any of the data centers updated across the globe in 1-2 seconds max.

Couchbase is only getting better

Having used Couchbase built from source (read community support only) since version 2.1 (Couchbase is now at 3.0.2), I can say that it is only getting better. They have made amazing progress with XDCR, added security functionality, and the N1QL language.

The Couchbase community is great. Checkout the IRC channel if you need help.

I Confess and Apologize, the really annoying ads – partially my fault.

sorry

You know those really annoying popup messages you get on your phone when you’re browsing? They aren’t easy to ignore like popups on your desktop. They are really, get in your face, make it hard to see the site you wanted to see, annoying. Well, I confess. I helped make those and I’m really sorry.

It was maybe two years ago and someone came to me with a new gimmick (That’s really all of AdTech summarized in one word: Gimmick ). It wasn’t the first time. I’d done a lot of work building affiliate marketing programs and ad servers. It was, however, possibly the most evil thing I have ever done and I apologize.

I tell myself that if I hadn’t done it, it would still have been done. I’m also sure that the company I built this for was not the only company to build it. AdTech is an industry flooded with companies doing exactly the same thing, all constantly copying the latest gimmicks from one another.

As a bit of penance, I offer to you the insider’s guide to what AdTech companies are selling you.

AdTech Gimmicks

There are four main categories of gimmicks:

  1. Tracking – These companies claim to (and possibly do) have some new and better way to know who saw their ads.
  2. Targeting – These companies claim deliver their ads to the people that you want to see them.
  3. Optimization – These companies claim to be able to determine which of your ads will perform better for the people seeing them.
  4. Positioning – These claim advertising real estate on the most expensive, well traveled properties on the Internet.

Most companies today will claim to give you at least 3/4 of the above to attract your business. If they don’t, they might suggest that you integrate their service into another to provide a more well rounded package.

Tracking

This is the ugly and dark, privacy invading, side of the business and the basis for almost everything else. The key to advertising is conversion and if you can’t connect the ad to the acquisition, you aren’t making money.

In the beginning, there were cookies. When people started denying or erasing cookies, there were super cookies – evil, twisted, perversions of technology that would store your tracking information in flash storage and rewrite that tracking information into your standard browser cookies at any chance. For some time now, AdTech has graduated to device fingerprinting (in addition to all of the above- that’s right- in many cases advertisers will use as many options as possible to make your acquaintance).

Recently, I read an article about over seven different methods for fingerprinting a user, many of which do not require cookies.

Targeting

Targeting is the next gimmick. The idea is that advertisers should show you ads that interest you. They do this by paying attention to the sites and content they see you browsing and connecting that information to your tracking id. If you browsed a site for men’s clothing, they should show you ads for men’s clothing.

Why is this a gimmick? The funny thing about targeting is it comes in at least two varieties: targeting and re-targeting.

Re-targeting more or less means showing you ads for sites you have already been to. The theory is that if you didn’t buy the first time, I should keep pounding my brand into your subconscious until you come back and buy something.

The reality of re-targeting is that you basically have an equal chance of seeing ads for something you decided you would never buy or seeing ads for things you already bought and are not going to buy again in the near future.

When the first happens, the ad companies call it targeting and that’s what their system is supposed to do- get you to buy something you haven’t bought. When the latter happens, they call it re-targeting. You already bought something, there is a great chance that you will buy something else.

Either way, it is a feature that they do better than everyone else. Definitely not a bug.

Optimization

Studies have shown that even experts with years of experience have trouble making accurate predictions about user behavior. For that reason, marketers have turned to science to decide what users like best.

On the simplest level, that means showing two or three versions of an ad (A/B testing) and after some time deciding to show only the version that got the most desired responses. On a more complicated, gimmicky, level companies will use a proprietary version of a Contextual Multi-Armed Bandit algorithm to pick which ads to show you.

While there is some real science and mathematics behind all this, and each company will have a PHD if not several to stand behind their algorithm, the facts of life are that we don’t live in a vacuum. What worked yesterday, because J Lo tweeted X on American Idol, will not necessarily work today when Amazon is not serving your ads in under 300ms. There is simply no way to prove that these algorithms are working in real life.

As a result, AdTech companies will push their optimization technology like there is no tomorrow. If you try them and get good results, all credit will go to their amazing tech. If things don’t go well, they will blame it on one of a hundred factors which they couldn’t control and maybe you will move onto another provider who will also have a 50/50 chance of getting your business with a similar gimmick.

Positioning

Positioning is one of the tried and true practices in advertising. Since the dawn of the billboard, putting your ad where more people will see it is the best way to get more customers, regardless of your conversion rates.

“How could that be a gimmick?”, you ask. On the simplest level you have the popups, the pop-unders, the transition ads, drive-by downloads and on mobile, the ever annoying and unescapable alert box. These are all positioning gimmicks. I’ll be there on top of your content. I’ll be there when you close your content. I’ll let you see your content in a couple seconds. I won’t let you see your content.

There are some more complicated plays on the positioning gimmick with the media exchanges and real time bidding (RTB). The idea behind them is simple. A site has demand for an ad in a certain location. They put that demand up for auction for exactly 100ms. Whoever bids the most inside 100ms gets their ad shown to the user.

Theoretically, when the demand is put up for auction, all sorts of information (tracking id, targeting information, etc.) is put up with it. Your favorite AdTech company will tell you that they have partnerships with all the best exchanges and that is the only way to get your content into the best positions.

In reality, everyone wants to be on the premium sites whether the ads are targeted or not so there is no point in using RTB there. The AdTech companies just buy impressions outright and split them between their campaigns (even if you think they are using RTB to get you there).

They will also tell you that their super algorithms (see optimization) will get you the best positioning for the best price and targeted at your users. It would be awesome if it worked.

  1. Even when there is RTB involved, the traffic is mostly lower quality traffic which just helps the AdTech companies beef up their click through ratings.
  2. There is no proof that the algorithms, supposedly buying only the impressions you want for the least amount of money, are working at all.

Buyer beware

They say “The proof is in the pudding” (originally “The proof of the pudding is in the eating”). It means that you don’t know if something is good until you try it.

If any of these AdTech companies really worked, would there be so many of them? Wouldn’t the super algorithm have devoured all the ad spaces on the Internet by now?

In my opinion, everyone is eating the same, mediocre pudding, these days and no matter what combination of the four AdTech gimmicks they try to push on you, don’t be afraid to be skeptical. Make them explain and prove why they are better (showing with a test campaign is not explaining or proving anything).

If you are planning on opening a new AdTech business yourself, please reconsider. There are many other areas of technology which could be measurably improved upon. For the most part, we are all using AdBlock Plus anyway.