Tag Archive for Israel

Ynet on AWS. Let’s hope we don’t have to test their limits.

tightrope

In Israel, more than in most places, no news is good news. Ynet, one of the largest news sites in Israel, recently posted a case study (at the bottom of this article) on handling large loads by moving their notification services to AWS.

“We used EC2, Elastic Load Balancers, and EBS… Us as an enterprise, we need something stable…”

They are contradicting themselves in my opinion. EBS and Elastic Load Balancers (ELB) are the two AWS services which fail the most and fail hardest with multiple downtimes spanning multiple days each.

EBS: Conceptually flawed, prone to cascading failures

EBS, a virtual block storage service, is conceptually flawed and prone to severe cascading failures. In recent years, Amazon has improved reliability somewhat, mainly by providing such a low level of service on standard EBS, that customers are default to paying extra for reserved IOPS and SSD backed EBS volumes.

Many cloud providers avoid the problematic nature of virtual block storage entirely, preferring compute nodes based on local, direct attached storage.

ELB: Too slow to adapt, silently drops your traffic

In my experience, ELBs are too slow to adapt to spikes in traffic. About a year ago, I was called to investigate availability issues with one of our advertising services. The problems were intermittent and extremely hard to pin down. Luckily, as a B2B service, our partners noticed the problems. Our customers would have happily ignored the blank advertising space.

Suspecting some sort of capacity problem, I ran some synthetic load tests and compared the results with logs on our servers. Multiple iterations of these tests with and without ELB in the path confirmed a gruesome and silent loss of 40% of our requests when traffic via Elastic Load Balancers grew suddenly.

The Elastic Load Balancers gave us no indication that they were dropping requests and, although they would theoretically support the load once Amazon’s algorithms picked up on the new traffic, they just didn’t scale up fast enough. We wasted tons of money in bought media that couldn’t show our ads.

Amazon will prepare your ELBs for more traffic if you give them two weeks notice and they’re in a good mood but who has the luxury of knowing when a spike in traffic will come?

Recommendations

I recommend staying away from EC2, EBS, and ELB if you care about performance and availability. There are better, more reliable providers like Joyent. Rackspace without using their cloud block storage (basically the same as EBS with the same flaws) would be my second choice.

If you must use EC2, try to use load balancing AMIs from companies like Riverbed or F5 instead of ELB.

If you must use ELB, make sure you run synthetic load tests at random intervals and make sure that Amazon isn’t dropping your traffic.

Conclusion

In conclusion, let us hope that we have no reasons to test the limits of Ynet’s new services, and if we do, may it only be good news.

Google Analytics fixed but is Google crashing?

Google has finally added Israel to the list of Countries in the sign up process which is good news.
On the other hand, they got the timezone wrong (Israel is in DST right now and uses GMT+3 till about October) and that’s after spending over a week fixing it.

What’s going on inside Google? Why did it take so long? Why wasn’t Israel on the list to begin with?
I recieved no explanation from Google but my guess is that they must of had a bug in the code generating the form fields and the javascript behind them. Look at the following code sample:

CC["ID"] = new Array("220|(GMT+07:00) Jakarta","234|(GMT+08:00) Makassar","221|(GMT+09:00) Jayapura");
CC["IR"] = new Array("257|(GMT+03:30) Tehran");
CC["IQ"] = new Array("198|(GMT+03:00) Baghdad");
CC["IE"] = new Array("300|(GMT+00:00) Greenwich Mean Time");
CC["IL"] = new Array("222|(GMT+02:00) Jerusalem");

Before the fix, Jerusalem time was present in the javascript but it was ORed to something else like the first line above.

That is still no excuse for such a system going live. Google’s quality control should have stepped in.

On the other hand it points to a growing list of technical difficulties within Google.

  1. Since Google’s last update to their algorithms, they’ve been returning pages from sites of mine that haven’t been online in years.
  2. For over a week I’ve been experiencing problems with Gmail timing out.
  3. Blogger is less than responsive as always.
  4. Analytics, in the day that I’ve been using it, has often claimed to be under maintenance one second and fine the next- I guess maintenance means “I’m a tired server, leave me alone please.”

The Register reports that Google is choking on web spam: http://www.theregister.co.uk/2006/05/04/google_bigdaddy_chaos/

Webmasters now report sites not being crawled for weeks, with Google SERPS (search engine results pages) returning old pages, and failing to return results for phrases that used to bear fruitful results.

“Some sites have lost 99 per cent of their indexed pages,” reports one member of the Webmaster World forum. “Many cache dates go back to 2004 January.” Others report long-extinct pages showing up as “Supplemental Results.”

But the new algorithms may not be solely to blame. Google’s chief executive Eric Schmidt has hinted at another reason for the recent chaos. In Google’s earnings conference call last month, Schmidt was frank about the extent of the problem.

“Those machines are full,” he said. “We have a huge machine crisis.”

While here they attempt to save face for Google by putting Schmidt’s comment in context, it’s clear that Google has been having technical problems.

Google continued to make substantial capital investments, mainly in computer servers, networking equipment and its data centers. It spent $345 million on such items in the first quarter, more than double the level of last year. Yahoo, its closest rival, spent $142 million on capital expenses in the first quarter.

Referring to the sheer volume of Web site information, video and e-mail that Google’s servers hold, Schmidt said: “Those machines are full. We have a huge machine crisis.”

Jordan Rohan of RBC Capital Markets called Google’s capital spending “unfathomably high,” noting that it spent the same percentage of its revenue on equipment as a wire-line phone company.

I don’t see how the context makes things any better. The bottom line remains that Google needed a heck of a lot more hardware than it had and who knows if they bought everything they needed. Those are only the first quarter figures- I would imagine it could take a whole quarter to deploy $345 million dollars of equipment. I wonder what they will spend next quarter?

I cried in Room #617

I just got this in an email.
Warning: Very emotional


I Cried in Room #617
By Tamar Yonah April 24, 2006

It was supposed to be a happy occasion. We were celebrating my son’s
Bar Mitzvah that fell in the week of Passover, and since we knew we
wouldn’t be able to host all the extended family in my house for a
Passover Seder and a Bar Mitzvah Sabbath, we came to the conclusion
that we would find an affordable hotel that would be able to sleep
and feed the family here in Israel, and those coming from abroad.
All was well and good. We arrived at a Jerusalem hotel, where I got
the keys to room 617. We excitedly took the elevator up to where our
new lodgings would be for the next few days. After glancing around
the room, I sent my kids and family down to the dining area where
they had a light meal offered before the Passover Seder. I stayed
behind to unpack their holiday clothes and to make up the beds for
the children.

I liked having some quiet after the brief chaos. We were 8 people
crammed into two small hotel rooms with an adjoining door. There was
lots of luggage, and little space to spread out. Two of my kids
would have to sleep on the floor, I decided. I took up the two
cushions from the couch bed that was in our room, and decided to
make it a mattress. Then leaning over to pull out the couch-bed, I
spotted a folded piece of paper between the mattress and the metal
support springs.

“Hmmm, what’s this?” I asked myself. “What a strange place to store
paper”. Taking it out and unfolding it I found it was actually two
pages, stapled together, typewritten, in Hebrew. I began to read it.

“Kinah” was the title (Kinah means ‘lamentation’ in English). No
author’s name. I started to read, and after the first three lines, I
took some weak steps backwards, fell onto the main bed of the room
and started to cry.

It was a poem someone had written. Someone who was living in my
hotel room before I got there. Someone who for 8 months was crammed
into this room with his family, bewildered, hurt, and forgotten by
his countrymen. He or she, had written a poem, a poem of their pain
and anguish, and I happened to find it. Was it left for me,
the ‘next occupant’ purposely? Or was it forgotten in the rush to
evacuate the hotel to let new guests take over even this temporary
abode for Passover?

I had heard that many of the families ensconced in these Jerusalem
area hotels for an “indefinite time”, were now being forced to move
again to make room for Passover and Easter guests. “Oh my gosh, this
room belonged to a family from Gush Katif!”. I was shaking. I read
the poem:

Kinah — a Lamentation / Dirge
(loose translation of the Hebrew below)

On the Expulsion of Jews from Gush Katif, The Gaza Strip and Northern Samaria, I will lament every year with a broken and bitter heart.

How is it possible that a Jewish Prime Minister — Ariel Sharon, can give away a land brimming with wisdom and talent to Arabs, oppressors of Israel, as a gift and present?

How is it possible that he cheated his voters by expressing his love saying, “In my days, you will sit safely on your land”, and then betrayed our trust, saying “To them it belongs (the Arabs) — not to you”?

How is it possible that people were sent to build, work and settle, affirm by self sacrifice the building of the land and now they were betrayed and disinherited, their land handed to Ishmaelites?

How is it possible that he made covenants with murderers and gave the land of G-d with no recompense, to rapists, and then he reproached and blasphemed, saying, there is no G-d in the ‘Land of Judges’ — ( the land of Israel.)?

How is it possible that he attempted to cover his, and his sons’, crimes with much cunning, and shed shame on those goodly ones who toil over Torah, -the people of The Land? They will concentrate on the disengagement, and his sons’ guilt will be lost in oblivion and turmoil.

How is it possible that the heads of Parliament, gave a majority to the decision — a parliament that was chosen by the nation…. to end the quiet dwelling of the old, and the playing of the children from their streets…. to expel them, shame them, and give what belongs to THEM, to a BLOOD THIRSTY nation?

How is it possible that he scorned the laws of the Torah, he and his company of friends, and he despised the lovely land and legislated laws of his own desire?

How is it possible that police, soldiers and judges all stood by his side and participated in the joys of all his evil deeds and did not say, “Enough. Our will is an evil one.”?

How is it possible that they announced to pioneering Jews settling the land: ‘Enough -Stop! You have already sat many years in your settlements and now we will give this land to your enemies — your oppressors’.

How is it possible that Israel, a holy nation, cried out in tears to undo and revoke the decree of insolents and criminals and G-d concealed His face and left His land to traitors?

How is it possible that we stood in prayer and supplication and poured out our hearts, perhaps G-d will spare His nation and remove the decree of an evil government and the nation of G-d will lament, for the land was given to the hand of the vain and evil?

How is it possible that Jews were forcibly expelled from their homes, tossed onto busses…. people, women, and their children and no mercy was shown for their property and their hard work?

How is it possible that they enacted a violent order…. police and soldiers, to remove their brothers — the sons of Israel — from their homes? Land, (Eretz) don’t cover their crimes until sins cease.

How is it possible that they destroyed out of hatred, houses, synagogues and schools full of Torah and mercilessly threw people out on the streets?

How is it possible that without any explanation or reason they destroyed factories, green houses and every growing field and gave their places to savages and the dishonest?

How is it possible that he raised his hand upon synagogues and yeshivot (Torah centers) and put an end to houses of study (batei midrash), and the chatter of infants, while saying:… Thus the world’s nations will love us more?

How is it possible that he invaded graves and extracted bones of holy ones killed by bullets, fire and stones and he did not feel he would be hit by a curse — shchik atzamot — where his bones are pulverized, ground , and pounded.’ ?

How is it possible that they gloated in the shadow of our sorrow, all his companions and lovers saying “G-d left the land, the house is empty of its Lord. Va-avdah B’haf -kaira nicha leya (in Aramaic) , Everyman should do as he sees fit.”?

He who has mercy on the poor…. (please) have mercy on the afflicted and return them to their legacy, for in You, they trust. And bring closer the redemption, for Israel shall be redeemed by G-d, for Eternity.
(End)

I finished the poem and I wanted to know…. Who were these people
who lived in room 617 for 8 months? How many children did they have?
What were their jobs before they were sacrificed by the politicians
who came to power? Were they happy to leave the hotel for
a ‘caravilla’ (trailer home), or were they victimized once again,
but this time, too tired and too broken to fight? Did they leave
this poem purposely for the next occupant of the room to find, which
happened to be me, or was it left behind by accident? Why didn’t the
maid find it, everything was cleaned well before Passover, even the
carpets were recently shampooed, I could feel the dampness in the
rugs with my feet. Did G-d want me to find this, to share this cry
with others?

There was no author’s name or the name of the person who it belonged
to.

Yes, I cried in room 617, I wonder how much crying there was in this
room the last 8 months before I got there.

Since the expulsion, I have prayed every morning that Hashem (G-d)
should bless our brothers and sisters from Gush Katif and the
Northern Samarian region, with even more wonderful and happier lives
than before and that things should start improving immediately for
them.

Crying is good. It’s not a solution, although, it’s a good start to
bring us to action. On behalf of our brothers and sisters who were
wrenched from their homes, lives and communities, please, let us all
do a special act of kindness in their merit. One can choose to:

1) Pray for their well being in your morning or evening prayers
everyday until they are settled.
2) Make a donation directly to these families to help them re-build
their lives
3) Visit them, write them or make a ‘twin city’ with them to show
that you care.
4) Make a firm commitment to act in the future, to prevent anything
like this from happening again.

May any future crying in room 617 be from cries of joy from new
occupants who have come to visit Israel to herald in the Moshiach
(Messiah) and help bring in the redemption that G-d has promised us
all. May we all cry from joy and not from sorrow. And, may we start
these positive actions now, to ensure it comes speedily in our days,
Amen.

If people want to give donations, I can put them in contact with the right people.
Write an email to gushkatif **at** yonahruss **dot** com.