Arrow Pos- Entire System Down

Our system went down around noon today and we called to get tech support, after an hour of trying to get through I find out that the whole Arrow network is down prevent all Arrow POS users from accessing their software. Why would them being down have any effect on my store? Are we constantly connected to them? If so, that seem really odd.

im down too. took forever to get through (which i understand) but the woman answering the phone was very rude(which i dont understand).i asked if she knew when i would be able to use my pos again she said 15 minutes(an hour ago) and hung up. Looks like pen paper and a calculator for a while.

I find it VERY odd that a problem at the company would impact your POS. What happens if your internet connection goes down? Does that stop your POS from working too?..

Looking at the activity log under the global settings, it appears the server pings our system every 7th minute of the hour (12:07, 1:07, etc) to see if our license is valid ("LICENSING: successfully validated licensing information). Everything was fine until 12:07 when the log said “LICENSING: The store’s customer ID is not valid” follwed by a message at 12:08 that said “LICENSING: The expiration date has passed”. I’m assuming part of the script written for everyone’s POS had today’s date at Noon as the expiration date, or it’s just something completely unrelated.

Why would a license expire? When your service contract is up I can see not getting upgrades and support, but generally one PURCHASES a license…

I think I would block them from accessing the machine without someone at the desk allowing it in each instance since they seem to be a little ragged on the details.

If I were in the market for a POS, this feature would really turn me off on this company. :shock:

Quickbooks recently hosed me in a similar way. They quit supporting version 2008, which I was aware of, they told me I would not recieve any new payroll updates untill I updated QB, but what they did not tell me was that they killed the tax tables in my system so when I ran payroll it did not withhold state or federal income tax. That upset me since I had paid for that data with my subscription, but not nearly as much as shutting down my whole operation for a day would. You guys that have that system need to demand a change to the program so this does not happen again.

This is Barry from ArrowPOS. I watch the various social marketing sites like this, and have never chimed in to promote our product. Arrow stands on its own merit. However, this instance is another matter altogether since it deserves explanation and apology to our customers and friends.

It was a nice quiet Saturday afternoon, and then it wasn’t… Our tech phones lit up, our call center got swamped, as did all of our personal cell phone’s text messages and voicemails. Like any POS company, there are the hardware issues that arise, particularly when we upgrade someone from another system and have to use their legacy equipment. Software is another issue as we simply don’t have many Arrow software issues, and when we do, they come from Beta sites and are addressed before we release to the masses. This was one of those big time WTH moments. Something was wrong and we pulled guys out of weddings, off lakes, and rallied the troops into the office to get it resolved.

It was an agonizing wait as the programmers went to work deciphering the issue. There were a number of us logged in and prepared to get our customers up as soon as the issue was resolved. In the end, it turned out that the company who houses our information did an update on Saturday morning and changed a simple thing on what is called a connection string. We had never seen the issue before, and once the issue was detected and resolved, we had everyone up within 45 minutes.

Now to address the very valid concerns posted above in this thread:

  1. Internet: One of the advantages of developing Arrow with the latest tools and platforms is the ability to incorporate and expand advanced internet functions never available before in any POS system. For instance, we use the internet for credit card processing, mapping, integrated loyalty and rewards, emailing, texting, and tech support. Many POS systems are glorified cash registers, and our vision is much greater than that.
  2. We Check for Internet Integrity: This is a must and called pinging. As noted above, we check about every 10 seconds and log the activity for reference in the event we get a tech call. By using the internet, we are also reliant on our client’s local internet providers. Not all providers or services are created equal, particularly in rural areas, and have intermittent outages. Most outages are very, very brief and undetectable to the casual user. We ping to make sure the internet is alive to prevent such things as internet time outs which will slow a system down or cause it to pause. If Arrow detects the internet is unavailable it reacts accordingly.
  3. What do we ping: Arrow reaches out to our servers to verify the internet is alive, searching for Arrow Updates, and potential piracy threats. Piracy and the protection of trade secrets are a very real threat to any POS company, whether by competitors or cloning by end users. I find it both comical and flattering that one of our primary competitors even had notebooks of our screen shots in their booth at the latest Pizza Expo.
  4. If the Local Internet Goes Down Altogether: Only the internet dependent features are affected. Arrow is not internet dependent to function.
  5. Security: This was not a security breach by any stretch.
  6. Action Plan: The issue was fixed, and certain areas of the code are being rewritten. It was a short term problem and has been addressed.

Once again, we sincerely apologize for any inconvenience to our customers. If you would like to chat, please email me at Barry@ArrowPOS.com.

Regards,
Barry Brownhill
President, ArrowPOS, Inc.

Barry,

That was an outstanding post. I have great respect for any business or business person that is willing to stand up and be counted when things go wrong as well as when they are going right.

I hope you feel welcome to participate on the forum. I am sure that there are many topics you could contribute to without crossing the line of self promotion.

Best of luck… and make sure you take care of those people you called in on a day off!

Steve

Hey Steve, thanks for the vote. We work really hard to keep our clients happy, and shooting ourselves in the foot is painful in many aspects. Worst of all, is any event that potentially risks losing the trust and respect of the people we serve. We are in a long term relationship business, some spanning 20 years and across generations of owners.

The beauty of open forums like the Think Tank is all companies are now held to a higher standard for performance. We welcome the challenge, as painful as it may be at times.

Thanks for listening,

Barry

I to have a ARROW POS system and it also went down on saturday, I called ARROW and they explained that something went wrong system wide and that they are working on getting it restore…I have known the folks at ARROW for awhile and had complete faith the problem quickly as possible and it was…
I have to say that the customer support/service at ARROW is outstanding and have always resolved any problem I ever had quickly…Thanks ARROW and keep up the great work…

WOW, our shop has used Arrow for two years and always found them to be quick when we required service. We were also down Saturday afternoon but felt like it was resolved quickly for such a large interuption. I read some of the complaints and just felt I needed to respond. What if you had a power outage, is that different? We could at least still operate! How about an oven break down? Some times it just hits the fan and we have to deal with it. Would you want your customers to go off on you when you get in the weeds and can not give your normal service. Lets let some time pass and cool off, Arrow has been very reliable and should not be blasted for what amounts to .003% of your year.

Nice post post Barry. Thank you for being a POS company with integrity and not just demanding that complaints about your POS just be removed.

Apparently not too odd and out of character in this situation as this sentiment was echoed by another Think Tank user at the top of the thread. We all find ourselves in situation where $hit hits the fan, but there is no room for customers to be treated rudely. I’m not much of a fan of the other pizza magazine, but the opening commentary in this months issue is a good read. One short quote from it: “Things happen, after all. Orders get lost. Delivery drivers get lost. But customer service should never get lost. That’s not a variable-- it’s a basic component that is critical to success.” Customers understand that you will not always be at your best, but how you handle yourself in those situations will dictate whether they continue to be your customers.

crusher and i are very much in the same boat. however my system was down for about 6.5 hours, and it was in the middle of our rush. also after it was up it went down again 45 min later for about an hour. this is what makes me concerned about the reliability of arrow. I have had many hats in this industry and the one im wearing now is that of a roaming gm for a franchise(basically that means i go to whichever store needs me). Actually our logo is on the front page of the arrow site. The reason i say this is because not all of our franchisee’s use arrow. I was treated very poorly by the call desk. This may be petty but i was also annoyed that barry’s first response was to a positive comment and not to crusher or i who are obviously his customers. Also i was curious if crusher had heard from any competing pos systems. The first and only call i received was from a comnpetior who had heard about the crash and immediatly tried to sell his service. not that i would switch so hastily but i find it odd that arrow would let another company in the door like that. like barry said they shot themselves in the foot, but it should not be my job to bandage that foot(ie. I email THEM if i have concerns). you pulled the trigger on this accident and it should be you doing the bandaging.

I think we all agree that no matter how fast any service interruption is…it is never restored fast enough. Arrow was hit by what I am guessing to be thousands of phone calls in a matter of minutes. I am not trying to justify how their customer service people responded to this, but I am sure the level of frustration was off the chart. I am also going to believe that not all the callers were polite on their own. Problems will happen from time to time and it does seem that they did what they could and as fast as they could to resolve the issue. On the other side of this, it should also be a good learning event for all the users that went down. A time to make sure that you have procedures in place to help get through any type of outage. I understand that the process will not be as efficient as having everything working in sync but you should still get food out the door. Maybe having a quick index of cards that help walk employees through what to do with any type of outage. Gas, power, phone, internet, ovens, cooler, POS, credit card processor, etc… there are so many things that can go wrong… being prepared is part of the plan. Having a writen policy on how to handle it all is helpful too. People work better having a reference to look at in any type of stressful situation.

It will be interesting for Arrow to pull the tapes from their call center and listen to some of those calls…

Za4life and Crusher, I do sincerely apologize to you guys and meant no disrespect. And Please, drop me a line. My first response was an open apology to all our customers and an honest explanation as to what happened, certainly not an excuse or a denial. The anonymity of this forum makes it hard to decipher who I am talking to and I was hoping that anyone who wanted to discuss it further would email me personally with their contact info to call them.

Interesting you should bring up the competitor… Yes, quite a number of our customers were contacted by a competitor who obviously went through our website for the testimonials and logos. More than that, while this was going on Monday morning, I got a strange call from a guy in their area claiming to be opening a pizza shop and wanted information, plus our “account list.” Coincidence? Maybe, but the timing and request were really odd.

The reports from our customers was the competitor was spreading some pretty wicked, head scratching, scare tactic misrepresentations in an obvious attempt to cast doubt on our software and our company. Take this for what its worth… from what I have learned from various sources, this company will systematically pounce on situations like this, and they will also respond to any press release about a franchise naming a POS partner. They will start calling all of the franchisees in a last ditch effort to undermine the relationship. Allegedly, the program is called “Intervention.”

Royster is right. The tone of the calls was all over the map. We are used to even the most heated and will typically let the customer vent, hopefully calm down, and then be ready to work through whatever issue prompted the call. Its part of the landscape of working with passionate people. Instead of voice mail, we use a call center with a live person to back up our phones, and they may handle a handful of calls a day. In the period of the first hour on Saturday, they had to handle close to 800 calls. Keep in mind, they were blindsided and had no concrete answers. The best they could do was say something along the lines of, “we understand and Arrow is working on it.” In the first half hour, they must have been thinking, “WTH!!!”

Even so, hanging up is unacceptable. The only time they are warranted to do so is if the call is threatening in nature, or a profanity laced tirade.

Regards,

Barry

It still seems VERY odd to me that a problem at the POS company would shut down customers systems. They should not be dependant on the company for anything relating to ordinary internal operations. This seems like a serious flaw in the overall system.

It would make sense to me, perhaps, if the online ordering sytem were impacted… but regular free standing operation of the POS in the store should be firewalled from headaches that originate outside the store.

I can see that, but it is not the case nor the intention. As discussed, Arrow is not dependent on internet to run. We encountered a really obscure quirk relating to a specific set of unique circumstances revolving around some older code. It has been addressed and removed.

Does anyone in their store have that one oddball PITA employee who knows more about computers than Bill Gates, and can’t keep his hands off the system? Very early on in development, we had a really odd incident where a customer had a “techie” driver who continually asked some very curious questions. After awhile, we determined it was highly likely he was trying to pirate the software. We have since found a better way to prevent piracy.

Although I was pretty upset with the situation, I understand that issues come about. I will still keep my Arrow system and in fact I just emailed them for a quote for a system for a 2nd store I’m looking at. I had another system before I upgraded to Arrow and had a ton of issues with it, arrow has been pretty good. The only problem I had was this one and it was a big one. I shouldn’t let that issue that was a true mistake cloud my impression of Arrow in general. I wouldn’t want my customers to stop doing business with me because of one bad experience.