Author: Yonah Russ

SmartDataCenter, the Open Cloud Platform that Actually Already Works

sdc

For years enterprises have tried to make OpenStack work and failed miserably. Considering how many heads have broken against OpenStack, maybe they should have called it OpenBrick.

Before I dive into the details, I’ll cut to the chase. You don’t have to break your heads on cloud anymore. Joyent have open sourced (as in get it on Github) their cloud management platform.

It’s free if you want (install it on your laptop, install it on a server). It’s supported if you want. Best of all, it actually works outside of a lab or CI test suite. It’s what Joyent runs in production for all their public cloud customers (I admit to being one of the satisfied ones). It’s also something they have been licensing out to other cloud providers for years.

Now for the deep dive.

What’s wrong with OpenStack?

First off, it isn’t a cloud in a box, which is what most people think it is. In 2013,Gartner called out OpenStack for consciously misrepresenting what OpenStack actually provides:

no one in three years stood up to clarify what OpenStack can and cannot do for an enterprise.

In case you’re wondering, the analyst also quoted Ebay’s chief engineer on the true nature of OpenStack:

… an instance of an OpenStack installation does not make a cloud. As an operator you will be dealing with many additional activities not all of which users see. These include infra onboarding, bootstrapping, remediation, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasting and management, billing or chargeback, reclamation, security, firewalls, DNS, integration with other internal infrastructure and tools, and on and on and on. These activities are bound to consume a significant amount of time and effort. OpenStack gives some very key ingredients to build a cloud, but it is not cloud in a box.

The analyst made it clear that:

vendors get this difference, trust me.

Other insiders put the situation into similar terms:

OpenStack has some success stories, but dead projects tell no tales. I have seen no less than 100 Million USD spent on bad OpenStack implementations that will return little or have net negative value.Some of that has to be put on the ignorance and arrogance of some of the organizations spending that money, but OpenStackÔÇÖs core competency, above all else, has been marketing and if not culpable, OpenStack has at least been complicit.

The motive behind the deception is clear. OpenStack is like giving someone a free Ferrari, pink slip and all but keeping the keys. You get pieces of a cloud but no way to run it. Once you have put all your effort into installing OpenStack and you realize what’s missing, you are welcome to turn to any one of the vendors backing OpenStack for one of their packaged cloud platforms.

OpenStack is a foot in the door. It’s a classic bait and switch but even after years, no one is admitting it. Instead, blue chip companies fight to steer OpenStack into the direction that suits them and their corporate offerings.

What’s great about SmartDataCenter?

It works.

The keys are in the ignition. You should probably stop reading this article and install it already. You are likely to get a promotion for listening to me ­čśë

Great Technology

SmartDataCenter was built on really great technologies like SmartOS (fork of Solaris), Zones, ZFS, and DTrace. Most of these technologies are slowly being ported to Linux but they are already 10 years mature in SDC.

  • Being based on a fork of Solaris brings you baked in enterprise ready features like IPSEC, IPF, RBAC, SMF, Resource management and capping, System auditing, Filesystem monitoring, etc.
  • Zones are the big daddy of container technology guaranteeing you the best on-metal performance for your cloud instances. If you are running a native SmartOS guest, you get the added benefit of CPU bursting and live machine re-size (no reboot, or machine pause necessary).
  • ZFS is the most reliable, high performance, file system in the world and is constantly improving.
  • DTrace is the secret to low level visibility with zero to no overhead. In cloud deployments where visibility is usually close to zero, this is an amazing feature. It’s even more amazing as the cloud operator.

Focus

SDC was built for one thing by one company, to replace the data centers of the past. It says so in the name. With one purpose, SDC has been built to be veryopinionated about what it does and how it does it. This gives SDC a tremendous amount of focus, something sorely lacking from would-be competition like OpenStack.

Lastly, it works.

Couchbase is Simply Awesome

couchbase

Here are five things that make Couchbase a go-to service in any architecture.

Couchbase is simple to setup.

Keep It Simple. It’s one of the axioms of system administration. Couchbase, though complicated under the hood, makes it very simple to setup even complicated clusters spanning multiple data centers.

Every node comes with a very user friendly web interface including the ability to monitor performance across all the nodes in the same machine’s cluster.

Adding nodes to a cluster is as simple as plugging in the address of the new node after which, all the data in the cluster is automatically rebalanced between the nodes. The same is true when removing nodes.

Couchbase is built to never require downtime which makes it a pleasure to work with.

If you are into automation a la chef, etc., Couchbase supports configuration via REST api. There are cookbooks available. I’m not sure about other configuration management tools but they probably have the relevant code bits as well.

Couchbase replaces Memcached

Even if you have no need for a more advanced NoSQL solution, there is a good chance you are using Memcached, Couchbase is the original Memcached on steroids.

Unlike traditional Memcached, Couchbase supports clustering, replication, and persistence of data. Using the Moxi Memcached proxy that comes with Couchbase, your apps can talk Memcached protocol to a cluster of Couchbase servers and get the benefits of automatic sharding and failover. If you want, Couchbase can also persist the Memcached data to disk turning your Memcached into a persistent, highly available key value store.

Couchbase is also a schema-less NoSQL DB

Aside from support for simple Memcached key/value storage, Couchbase is a highly available, easy to scale, JSON based DB with auto-sharding and built in map reduce.

Traditionally, Couchbase uses a system called views to perform complicated queries on the JSON data but they are also working on a new query language called N1QL which brings tremendous additional ad hoc query capabilities.

Couchbase also supports connectivity to Elastic Search, Hadoop, and Talend.

Couchbase is all about global scale out

Adding and removing nodes is simple and every node in a Couchbase cluster is read and write capable all the time. If you need more performance, you just add more nodes.

When one data center isn’t enough, Couchbase has a feature called cross data center replication (XDCR), letting you easily setup unidirectional or bidirectional replication between multiple Couchbase clusters over WAN. You can even setup full mesh replication though it isn’t clearly described in their documentation.

Unlike MongoDB, which can only have one master, Couchbase using XDCR allows apps in any data center to write to their local Couchbase cluster and that data will be replicated to all the other data centers.

I recently setup a system using five Couchbase clusters across the US and Europe, all connected in a full mesh with each other. In my experience, data written in any of the data centers updated across the globe in 1-2 seconds max.

Couchbase is only getting better

Having used Couchbase built from source (read community support only) since version 2.1 (Couchbase is now at 3.0.2), I can say that it is only getting better. They have made amazing progress with XDCR, added security functionality, and the N1QL language.

The Couchbase community is great. Checkout the IRC channel if you need help.

I Confess and Apologize, the really annoying ads – partially my fault.

sorry

You know those really annoying popup messages you get on your phone when you’re browsing? They aren’t easy to ignore like popups on your desktop. They are really, get in your face, make it hard to see the site you wanted to see, annoying. Well, I confess. I helped make those and I’m really sorry.

It was maybe two years ago and someone came to me with a new gimmick (That’s really all of AdTech summarized in one word: Gimmick ). It wasn’t the first time. I’d done a lot of work building affiliate marketing programs and ad servers. It was, however, possibly the most evil thing I have ever done and I apologize.

I tell myself that if I hadn’t done it, it would still have been done. I’m also sure that the company I built this for was not the only company to build it. AdTech is an industry flooded with companies doing exactly the same thing, all constantly copying the latest gimmicks from one another.

As a bit of penance, I offer to you the insider’s guide to what AdTech companies are selling you.

AdTech Gimmicks

There are four main categories of gimmicks:

  1. Tracking – These companies claim to (and possibly do) have some new and better way to know who saw their ads.
  2. Targeting – These companies claim deliver their ads to the people that you want to see them.
  3. Optimization – These companies claim to be able to determine which of your ads will perform better for the people seeing them.
  4. Positioning – These claim advertising real estate on the most expensive, well traveled properties on the Internet.

Most companies today will claim to give you at least 3/4 of the above to attract your business. If they don’t, they might suggest that you integrate their service into another to provide a more well rounded package.

Tracking

This is the ugly and dark, privacy invading, side of the business and the basis for almost everything else. The key to advertising is conversion and if you can’t connect the ad to the acquisition, you aren’t making money.

In the beginning, there were cookies. When people started denying or erasing cookies, there were super cookies – evil, twisted, perversions of technology that would store your tracking information in flash storage and rewrite that tracking information into your standard browser cookies at any chance. For some time now, AdTech has graduated to device fingerprinting (in addition to all of the above- that’s right- in many cases advertisers will use as many options as possible to make your acquaintance).

Recently, I read an article about over seven different methods for fingerprinting a user, many of which do not require cookies.

Targeting

Targeting is the next gimmick. The idea is that advertisers should show you ads that interest you. They do this by paying attention to the sites and content they see you browsing and connecting that information to your tracking id. If you browsed a site for men’s clothing, they should show you ads for men’s clothing.

Why is this a gimmick? The funny thing about targeting is it comes in at least two varieties: targeting and re-targeting.

Re-targeting more or less means showing you ads for sites you have already been to. The theory is that if you didn’t buy the first time, I should keep pounding my brand into your subconscious until you come back and buy something.

The reality of re-targeting is that you basically have an equal chance of seeing ads for something you decided you would never buy or seeing ads for things you already bought and are not going to buy again in the near future.

When the first happens, the ad companies call it targeting and that’s what their system is supposed to do- get you to buy something you haven’t bought. When the latter happens, they call it re-targeting. You already bought something, there is a great chance that you will buy something else.

Either way, it is a feature that they do better than everyone else. Definitely not a bug.

Optimization

Studies have shown that even experts with years of experience have trouble making accurate predictions about user behavior. For that reason, marketers have turned to science to decide what users like best.

On the simplest level, that means showing two or three versions of an ad (A/B testing) and after some time deciding to show only the version that got the most desired responses. On a more complicated, gimmicky, level companies will use a proprietary version of a Contextual Multi-Armed Bandit algorithm to pick which ads to show you.

While there is some real science and mathematics behind all this, and each company will have a PHD if not several to stand behind their algorithm, the facts of life are that we don’t live in a vacuum. What worked yesterday, because J Lo tweeted X on American Idol, will not necessarily work today when Amazon is not serving your ads in under 300ms. There is simply no way to prove that these algorithms are working in real life.

As a result, AdTech companies will push their optimization technology like there is no tomorrow. If you try them and get good results, all credit will go to their amazing tech. If things don’t go well, they will blame it on one of a hundred factors which they couldn’t control and maybe you will move onto another provider who will also have a 50/50 chance of getting your business with a similar gimmick.

Positioning

Positioning is one of the tried and true practices in advertising. Since the dawn of the billboard, putting your ad where more people will see it is the best way to get more customers, regardless of your conversion rates.

“How could that be a gimmick?”, you ask. On the simplest level you have the popups, the pop-unders, the transition ads, drive-by downloads and on mobile, the ever annoying and unescapable alert box. These are all positioning gimmicks. I’ll be there on top of your content. I’ll be there when you close your content. I’ll let you see your content in a couple seconds. I won’t let you see your content.

There are some more complicated plays on the positioning gimmick with the media exchanges and real time bidding (RTB). The idea behind them is simple. A site has demand for an ad in a certain location. They put that demand up for auction for exactly 100ms. Whoever bids the most inside 100ms gets their ad shown to the user.

Theoretically, when the demand is put up for auction, all sorts of information (tracking id, targeting information, etc.) is put up with it. Your favorite AdTech company will tell you that they have partnerships with all the best exchanges and that is the only way to get your content into the best positions.

In reality, everyone wants to be on the premium sites whether the ads are targeted or not so there is no point in using RTB there. The AdTech companies just buy impressions outright and split them between their campaigns (even if you think they are using RTB to get you there).

They will also tell you that their super algorithms (see optimization) will get you the best positioning for the best price and targeted at your users. It would be awesome if it worked.

  1. Even when there is RTB involved, the traffic is mostly lower quality traffic which just helps the AdTech companies beef up their click through ratings.
  2. There is no proof that the algorithms, supposedly buying only the impressions you want for the least amount of money, are working at all.

Buyer beware

They say “The proof is in the pudding” (originally “The proof of the pudding is in the eating”). It means that you don’t know if something is good until you try it.

If any of these AdTech companies really worked, would there be so many of them? Wouldn’t the super algorithm have devoured all the ad spaces on the Internet by now?

In my opinion, everyone is eating the same, mediocre pudding, these days and no matter what combination of the four AdTech gimmicks they try to push on you, don’t be afraid to be skeptical. Make them explain and prove why they are better (showing with a test campaign is not explaining or proving anything).

If you are planning on opening a new AdTech business yourself, please reconsider. There are many other areas of technology which could be measurably improved upon. For the most part, we are all using AdBlock Plus anyway.

Wrangling Elephants in the Cloud

elephant

You know the elephant in the room, the one no one wants to talk about. Well it turns out there was a whole herd of them hiding in my cloud. There’s a herd of them hiding in your cloud too. I’m sure of it. Here is my story and how I learned to wrangle the elephants in the cloud.

Like many of you, my boss walked into my office about three years ago and said “We need to move everything to the cloud.” At the time, I wasn’t convinced that moving to the cloud had technical merit. The business, on the other hand, had decided that, for whatever reason, it was absolutely necessary.

As I began planning the move, selecting a cloud provider, picking tools with which to manage the deployment, I knew that I wasn’t going to be able to provide the same quality of service in a cloud as I had in our server farm. There were too many unknowns.

The cloud providers don’t like to give too many details on their setups nor do they like to provide many meaningful SLAs. I have very little idea what hardware I’m running. I have almost no idea how it’s connected. How many disks I’m running on? What RAID configuration? How many IOPS can I count on? Is a disk failing? Is it being replaced? What will happen if the power supply blows? Do I have redundant network connections?

Whatever it was that made the business decide to move, it trumped all these unknowns. In the beginning, I focused on getting what we had from one place to the other, following whichever tried and true best practices were still relevant.

Since then, I’ve come up with these guiding principles for working around the unknowns in the cloud.

  • Beginners:
    • Develop in the cloud
    • Develop for failure
    • Automate deployment to the cloud
    • Distribute deployments across regions
  • Advanced:
    • Monitor everything
    • Use multiple providers
    • Mix and match private cloud

Wrangling elephants for beginners:

Develop in the cloud.

Developers invariably want to work locally. It’s more comfortable. It’s faster. It’s why you bought them a crazy expensive MacBook Pro. It is also nothing like production and nothing developed that way ever really works the same in real life.

If you want to run with the IOPS limitations of standard Amazon EBS or you want to rely on Amazon ELBs to distribute traffic under sudden load, you need to have those limitations in development as well. I’ve seen developers cry when their MongoDB deployed to EBS and I’ve seen ELBs disappear 40% of a huge media campaign.

Develop for failure.

Cloud providers will fail. It is cheaper for them to fail and in the worst case, credit your account for some machine hours, than it is for them to buy high quality hardware and setup highly available networks. In many cases, the failure is not even a complete and total failure (that would be too easy). Instead, it could just be some incredibly high response times which your application may not know how to deal with.

You need to develop your application with these possibilities in mind. Chaos Monkey by Netflix is a classic, if not over-achieving example.

Automate deployment to the cloud.

I’m not even talking about more complicated, possibly over complicated, auto-scaling solutions. I’m talking about when it’s 3am and your customers are switching over to your competitors. Your cloud provider just lost a rack of machines including half of your service. You need to redeploy those machines ASAP, possibly to a completely different data center.

If you’ve automated your deployments and there aren’t any other hiccups, it will hopefully take less than 30 minutes to get back up. If not, well, it will take what it takes. There are many other advantages to automating your deployments but this is the one that will let you sleep at night.

Distribute deployments across regions.

A pet peeve of mine is the mess that Amazon has made with their “availability zones.” While the concept is a very easy to implement solution (from Amazon’s point of view) to the logistical problems involved in running a cloud service, it is a constantly overlooked source of unreliability for beginners choosing Amazon AWS. Even running a multi-availability zone deployment in Amazon only marginally increases reliability whereas deploying to multiple regions can be much more beneficial with a similar amount of complexity.

Whether you use Amazon or another provider, it is best to build your service from the ground up to run in multiple regions, even only in an active/passive capacity. Aside from the standard benefits of a distributed deployment (mitigation of DDOS attacks and uplink provider issues, lower latency to customers, disaster recovery, etc.), running in multiple regions will protect you against regional problems caused by hardware failure, regional maintenance, or human error.

Advanced elephant wrangling:

The four principles before this are really about being prepared for the worst. If you’re prepared for the worst, then you’ve managed 80% of the problem. You may be wasting resources or you may be susceptible to provider level failures, but your services should be up all of the time.

Monitor Everything.

It is very hard to get reliable information about system resource usage in a cloud. It really isn’t in the cloud provider’s interest to give you that information, after all, they are making money by overbooking resources on their hardware. No, you shouldn’t rely on Amazon to monitor your Amazon performance, at least not entirely.

Even when they give you system metrics, it might not be the information you need to solve your problem. I highly recommend reading the book Systems Performance – Enterprise and the Cloud by Brendan Gregg.

Some clouds are better than others at providing system metrics. If you can choose them, great! Otherwise, you need to start finding other strategies for monitoring your systems. It could be to monitor your services higher up in the stack by adding more metric points to your code. It could be to audit your request logs. It could be to install an APM agent.

Aside from monitoring your services, you need to monitor your providers. Make sure they are doing their jobs. Trust me that some times they aren’t.

I highly recommend monitoring your services from multiple points of view so you can corroborate the data from multiple observers. This happens to fit in well with the next principle.

Use multiple providers.

There is no way around it. Using one provider for any third party service is putting all your eggs in one basket. You should use multiple providers for everything in your critical path, especially the following four:

  • DNS
  • Cloud
  • CDN
  • Monitoring

Regarding DNS, there are some great providers out there. CloudFlare is a great option for the budget conscious. Route53 is not free but not expensive. DNSMadeEasy is a little bit pricier but will give you some more advanced DNS features. Some of the nastiest downtimes in the past year were due to DNS provider

Regarding Cloud, using multiple providers requires very good automation and configuration management. If you can find multiple providers which run the same underlying platform (for example, Joyent licenses out their cloud platform to various other public cloud vendors), then you can save some work. In any case, using multiple cloud providers can save you from some downtime, bad cloud maintenance or worse.

CDNs also have their ups and downs. The Internet is a fluid space and one CDN may be faster one day and slower the next. A good Multi-CDN solution will save you from the bad days, and make every day a little better at the same time.

Monitoring is great but who’s monitoring the monitor. It’s a classic problem. Instead of trying to make sure every monitoring solution you use is perfect, use multiple providers from multiple points of view (application performance, system monitoring, synthetic polling).

These perspectives all overlap to some degree backing each other up. If multiple providers start alerting, you know there is a real actionable problem and from how they alert, you can sometimes home in on the root cause much more quickly.

If your APM solution starts crying about CPU utilization but your system monitoring solution is silent, you know that you may have a problem that needs to be verified. Is the APM system misreading the situation or has your system monitoring agent failed to warn you of a serious issue?

Mix and match private cloud

Regardless of all the above steps you can take to mitigate the risks of working in environments not completely in your control, really important business should remain in-house. You can keep the paradigm of software defined infrastructure by building a private cloud.

Joyent license their cloud platform out to companies for building private clouds with enterprise support. This makes a mixing and matching between public and private very easy. In addition, they have open sourced the entire cloud platform so if you want to install without support, you are free to do so.

Summary

When a herd of elephants is stampeding, there is no hope of stopping them in their tracks. The best you can hope for is to point them in the right direction. Similarly, in the cloud, we will never get back the depth of visibility and control that we have with private deployments. What’s important is to learn how to steer the herd so we are prepared for the occasional stampede while still delivering high quality systems.

Save Money on Private SSL CDN and Improve Performance at the Same Time

burning money

SSL support in your site, and therefore, in your CDN is critical but it is also incredibly expensive. In this article, I’ll show you how to save a fortune while improving the performance and security of your site.

It used to be that sites only encrypted the most sensitive traffic with their customers, i.e. registration, login, checkout, etc. In 2010, the Firesheep extension made it very clear that this is not enough.

Since then, many other attacks on partially encrypted sites have been devised and so major sites like Google, Twitter, Facebook, and others have all switched to HTTPS by default, aka Always On SSL. Last year, Google called for HTTPS everywhere and later, announced that secured sites will get boosts in PageRank.

Shared SSL from the CDN is cheap.

Shared SSL in the CDN is better than nothing but it isn’t great. It’s better than nothing because it will help with mixed content warnings. It’s not great because it isn’t on the same domain as your site so it can cause same origin policy problems.

Why is Private SSL CDN so expensive?

That’s a great question and I’m not sure there is a good answer. Some of the common excuses are:

  1. Encrypted traffic is heavier than unencrypted traffic so it should cost more. This is only very slightly true and recent work in this area has made this even less true.
  2. The logistics of deploying the SSL Certificates across all the CDN edge nodes is expensive. It’s honestly hard to believe that companies base all their business on managing uncountable edge nodes, have real trouble deploying the certificates.
  3. It’s only worth it for the CDN company to assume the security risk of HTTPS traffic if they charge you more money. This is probably the closest answer to the truth.

Regardless of the real reason, CDN’s often charge both exorbitant one time setup fees and high monthly fees for each SSL enabled configuration.

Common mistakes which are costing you money.

Companies often make several mistakes when ordering SSL services, costing them thousands of extra dollars each month:

  1. Ordering too many domains
  2. Ordering the wrong type of SSL service
  3. Not negotiating the fees

Ordering too many domains

Companies often spread their sites over multiple subdomains, i.e. www.yahu.com, mail.yahu.com, shopping.yahu.com, etc. Each of these sites will have it’s own content and many companies understandably setup separate CDN configurations for each origin, i.e. cdn.www.yahu.com, cdn.mail.yahu.com.

Ordering SSL from the CDN for each of those domains will cost a fortune. Instead, create a single CDN configuration for each root domain, i.e. cdn.yahu.com, cdn.giggle.com. In each configuration, use the CDN provider’s built in URL rewriting and origin rewriting features to direct the requests to the appropriate origin.

For example, configure cdn.yahu.com/www/ to cache content from www.yahu.com while cdn.yahu.com/shopping/ caches content from shopping.yahu.com. Now you have cut down the number of SSL slots you need to order drastically.

Ordering the wrong type of SSL service

Different CDN providers offer different types of SSL service. Some provide standard single domain certificates. Some use a multi-domain certificate. Some offer wildcard certificates.

Now that you have minimized the number of domains you need to protect in the CDN, you can re-evaluate the type of SSL certificate you need. In the example above, we might have thought to order an expensive wildcard certificate to protect all the subdomains of yahu.com, whereas now we can choose a less expensive single domain certificate.

If we can’t use URL rewriting to save on SSL enabled CDN domains, it may be less expensive to get one wildcard certificate, than to get several domains on a multi-domain certificate.

Not negotiating the fees

CDN fees are always negotiable. Usually, the more traffic you commit to each month, the lower a price you get. SSL slots are also open to volume discounts. If you need multiple SSL slots, try to order them at the same time and ask for a discount on the setup fees. The logic- even if they install each certificate manually, they can install all your certs at the same time.

You should also be able to get a discount on the monthly fees just because you are committing to pay them more each month. A better deal to ask for, is to offer to over-commit on monthly traffic commitment, in exchange for a discount on the SSL monthly fees.

For example, you have a monthly $5K commit for traffic and you are adding five SSL slots at a $750/month list price. They offer to go down to $700 monthly per SSL because you are adding five (a total of $8.5K monthly commitment). Counter back with an offer to commit to $6K monthly traffic + $500 monthly per SSL. This is better because, even if your traffic grows by 20%, you will continue paying $8.5K/month but you are still getting the SSL service.

How does this improve performance?

There is an added, hidden benefit in consolidating your CDN hostnames.

Now, when a customer first reaches one of your sites, i.e. www.yahu.com, their browser looks up and connects to your consolidated CDN domain – cdn.yahu.com. When they browse your site and switch to a new subdomain, for example if they clicked on shopping.yahu.com, they are already connected to cdn.yahu.com. They save on additional DNS lookups and additional TCP handshakes (the most expensive part of SSL traffic).

You can push this even further by sharing resource between your origins using a special CDN location, for example by storing common CSS files at cdn.yahu.com/shared/. In this case, shopping.yahu.com will be able to use the pre-loaded and cached CSS files from the browser’s initial visit to www.yahu.com.

Summary

I’ve used these SSL consolidation techniques in both Akamai and CDNetworks┬árealizing significant cost and performance savings. Although I’ve worked with at least five other CDN providers, I haven’t tried these techniques elsewhere.

If you’re interested in optimizing your CDN deployment for best cost/performance, feel free to contact me via LinkedIn or via┬áhttps://donatemyfee.org/.