Making sense of data this GOSF

The way consumers behave online today, there’s colossal amount of data that is getting generated and marketers are faced with making sense of this. Every time there’s an e-sales fest around the corner, this scenario gets more complicated. Here are some aspects to keep in mind as you plan for GOSF this year.

The ‘Window Shopping effect’:

As you know, traffic to your website increases 2 to 3 days ahead. These are trips buyers make to your website to plan shopping- typically looking at products they are interested in and the offers. We call this the ‘Window shopping effect’.

window shopping

While you don’t see any raise in sales during this period you have invaluable user intent information available with you. Make use of user behavioural data while planning your laying your strategy for the sales fest.

Unifying data:

Businesses today interact with the consumer through desktop and mobile websites, social media, apps, retail stores etc. The various touch points with the consumer generate tons of consumer behavior data, stored in different formats with different teams in your organisation. This data generally lies unused or under-used by the relevant functions. Bringing all this data together and mapping user profiles will help get a better insights.

How much is too much?

Sales fests is all about discounts and you know this best. Your buyers are out there looking for the best deal. Albeit, it is not the huge discount but the value savings that buyers look for. For example: here are some interesting facts that surfaced when a leading e commerce player in India had a festival sale earlier this year.

What happened with an E commerce company in a sales fest

Aggregate past purchase data, ad click data and products viewed to determine the offers and discounts and lure swarm of buyers to your website.

Keep the awesomeness alive this GOSF with Data-driven marketing!

5 Tips for A Kick-Ass Banner Ad

The holiday season has just begun and online shoppers have more reason to rejoice this year due to a clutter of deals and offers. This means competition. Read our blog on how to beat it. Retargeting comes in handy to stand out in the frenzy of ads out there.

With advanced technologies deployed in retargeting, you can personalize ads for buyers in real-time. Here are 5 key aspects that make a kick-ass banner ad.

Vizury Creative

While in Rome… Geography is one of the biggest contributors for the design. The regional and socio-cultural environment informs the preferences of the consumer. For example we see a much better appetite for a larger palette of colors and animations amongst Chinese buyers. We attribute the diverse color palette to the ones used in their religious and cultural setups.

Myntra

As you like it: Retargeting banners often work best when a combination of recommendations or choices are provided to the user. However the ways in which one recommends products are also specific for every industry. For e-commerce buyers, providing brand recommendations works better than product recommendations. But the reverse holds true for luxury brands.

Pink or Blue? Gender is another major determinant in terms of the kind of messaging, color palette and the nature of goods one can cross-sell/ upsell with the products browsed by the buyer.

No reasons attached: Buyers not only purchase out of need but also to accomplish a sense of discovery (very often seen with books, movies, songs etc) or complete a series of collectibles (very often seen with sports and historic memorabilia). In such cases there is little urgency except what is induced by the seller and a sense of closure is generally sought for such goods, making banner a key actor in the entire play.

Read between the lines: The point in the purchase funnel, called the ‘Stage’, that the customer drops-off at (eg. Landing page, product page, shopping cart etc) indicates his/her readiness to consume the product. For example: if user A has dropped off your website from the category stage for backpack indicates that he is evaluating his options. So a multi-product banner with 4 or more backpack recommendations would work. Whereas if he had added a backpack to his shopping cart shows he has made his decision. In such a case, showing an ad with just that backpack would make him come back and buy it. Hence, pertinent messaging helps bring a faster closure to the purchase cycle.

The future of personalization is at the intersection of customer transaction history, context and intent. Do your ads have the emotional hook to draw buyers? Tell us about it.

Hurry Christmas, Hurry Fast!

It is time for joy, cheer and a lot of magic. With shopping done, all of us hang our stockings and await our gifts. For a change, we sent something across to the all digital Santa and this letter fell through our chimney. It read…

My dear Vizury,

I had absolute fun with your pre-Christmas present. Here’s what happened when I found your magic box under my Christmas tree…

Love,

Santa

Santa- Final-221214

November in the Vizury Newsroom

 

November

It has been frenetic in the ‘newsroom’ for the last couple of months! Among many other things, our mobile retargeting product Vizury MobiConvert picked up on the action rapidly, we were recognized for our exemplary and steady growth, we integrated with the world’s most loved social network, we wrote some interesting and thought-provoking content, and our brand new website became a redesign case study! Phew!

So here’s the lowdown:

AdExchanger interviews our CEO, Chetan Kulkarni – Chetan was interviewed by AdExchanger, one of the world’s best known websites in the ad tech space. He says, “When I look at the channels for our business right now as it relates to exchanges, it is mostly display. We have robust RTB integrations with Google, AppNexus, Facebook, the Rubicon Project and all the usual suspects. In the end, the ad-exchange model helps our advertisers reach their users at a point in time…” – India-Based Vizury Targeting Digital CRM in Asia-Pacific Region by AdExchanger.

Mobile retargeting is à la mode – Our mobile retargeting product team now functions as a ‘company-within-a-company’ headed by our COO Gourav Chindlur. Vizury MobiConvert is revolutionizing the way marketers target mobile device users who have already visited the marketer’s mobile asset (website or app). And the rich results are kicking in already – see this infographic.

We are #2 on Deloitte Tech Fast 50, 2013, a list of the 50 fastest growing technology companies in based in India. Our CEO Chetan Kulkarni credits the company’s stupendous revenue growth over the past three years to the successful international expansion and the strengths of our key website retargeting offering, Vizury WebConvert. The list is available here.

Our first whitepaper is out – We used data from our global campaigns to analyse how retargeted users behaved differently in every industry. This formed the basis of our first whitepaper, “Industry Driven Retargeting: the benefits of a domain-driven retargeting strategy.” It also recommends several industry-specific features that advertisers should build into their retargeting campaigns.

We are now directly integrated with Facebook Exchange – We announced our direct integration with Facebook Exchange, Facebook’s real-time ad buying platform. This integration will enable us display ads on Facebook, in addition to existing display inventory.

Our website is a case studyOur brand new website which is based on flat UI design is now a case study! It has been described as a pleasant deviation from template style that is so ubiquitous today. “The fact that the above information heavy pages were consumed and that there has been a 100% increase in the time spent on those pages indicate phenomenal success.” – The complete reference for a Website revamp by Jumpcontent.

Lead Digital, Germany interviews Edvaldo Acir, our Latin America Head – Ed was interviewed by Lead Digital, one of the most influential German Digital Business Magazine. Ed says, “2013 was the biggest growth spurt over Mobile. Mobile applications are therefore – together with premium environments within web portals – the next step.” – The biggest growth spurt by Mobile by Lead Digital, Germany.

We would love to hear from you. We’ll be back soon. Till then, do say hello to us at marketing@vizury.com!

Affiliate Data Sharing Tips- Setting up a Framework for Sharing Data [PM Video Series 02]

 

“Affiliate data sharing” has been around for a while now, yet the buzz doesn’t fade away. Why?

As brands go on to work with a motley group of affiliates, managing their individual data needs turns nightmarish. Channel partners, online aggregators, ad partners, agencies – the list of affiliates is endless with their unique data needs. While a price comparison site might need product (catalog) data only, sophisticated channels like retargeting need both user-intent and product data.

Here’s a video that helps you understand and deal with all that can happen with affiliate data sharing.

Wondering why you need a framework or a process to manage data? Well, here is what could go wrong:

Data security: Safe-guarding personally identifiable information (PII) or other sensitive data needs to be a part of the brand’s DNA. Like disaster recovery solutions, everyone knows it is important but rarely is anything done till it’s too late.

User experience: If not done right, how you distribute data can impact other systems which may hamper users’ experience with your brand/website. For e.g. badly engineered affiliate tags could bring website page loads to a crawl.

Consistent and correct messaging: Imagine the after-taste when a user sees and clicks on an ad for a discount offer that actually expired days ago! Not great, right? Messaging is important and advertisers should moderate what affiliates say to potential customers.

Sub-optimal ROI: If affiliates aren’t enabled with the right data, advertisers are actually getting in the way of their delivering the best results they are capable of.

Working with hundreds of clients, we are no strangers to the aforementioned challenges. To make lives easier, we have come up with a handy three step framework to effectively visualize, monitor and enhance the data input setup. You begin with segregating the data then build a data dictionary to map the different data sources to your affiliates.

And, voila, the right set of people get the right set of information in the right frequency.

Analysis by Data Analytics- A Way to Predict Your Consumer’s Decision

The role of data sciences has evolved over time. What used to be an after-thought is now a multi-billion dollar industry that can make and break businesses. With the proliferation of wearable and the impending chip boom in Internet of Things (IoT), data sciences presents a cutting edge proposition for businesses to understand competition and consumers alike. Any product you conceptualize, it has to be supported with corresponding data. It is the very fabric of businesses now. As the common adage goes, a prototype is worth a thousand meetings and you cannot have the perfect prototype without the backing of data. If a digital company is not interpreting and reading data, it has a weak shelf life and it is probably going to disappear in months, if not years.

Data sciences can cut edges, no matter which sector a company is in, lead the charge in innovation and transform companies. The cornerstone of any technology business is built on data. Companies cannot live in isolation on the power of data. One has to embrace the efficacy of predictive analytics that can gauge sentiments and build brands. The ad technology and consumer internet companies were early adopters of data sciences. Without analytics, these companies could not compete in scale and understand demand. Telecom is also a big player and it is quickly digitizing their data. The compelling advantages of data is now being realized by all sectors especially Banking, Financial services and Insurance sector. Big data does not offer solutions; it poses problems and hypothesis for business. It is only through the science of data, namely, analytics, does one truly offer business decisions to management that can result in ROI. In India, the market for structuring data and deriving meaning from it is weak, however that is changing with the explosion of software-defined businesses.

With the government’s impetus in digital India initiatives, data sciences can help set up cloud-based education systems. The primary advantage of a cloud-based education system is that it can bridge the urban-rural divide. It can also tackle the paucity of teachers irrespective of geographical barriers. In the data sciences field, there is no formal course for a candidate to equip himself with advanced technologies. According to IDC, the big data and analytics market will reach $125 billion worldwide in 2015. However, yet there is no formal course that prepares candidates to tackle such problems. We look for engineers and statisticians who can read data, but there is a gaping hole with respect to the quality of education for this field. Most companies have their own labs to train and prepare candidates on the job, but imagine if candidates can already have a prerequisite set of skills at the basic level. The education sector has a huge role to play in enabling candidates to innovate and think out of the box for a niche field like data sciences.

The entire business runs on data. Any discussions start with proven records and samples. Any product that we conceive or how we approach ad inventory, margins, runs on data. By collecting data, a deep profile of a user builds and understands. This allows advertisers to reach the right kind of audience and assure consistent ROIs. The bedrock of this entire paradigm runs on data sciences and our ability to assure increased revenues for a customer.
In the past decade information Technology has grown in size and numbers. India is moving into an era where we are building software in India. As we see companies building technology based solutions for a range of verticals, the need for data scientists is going to increase. It is going to be one of the hottest jobs for the next decade or more. More exposure to course content in machine learning is needed. A unique blend of statistics and engineering skills would help candidates to explore untapped potential. We are going to see more entrepreneurs in India and with that growth, we will see more data scientists jostling and competing at the highest levels. With Sharpen your creative energies, learn to love numbers, read voraciously, with these three ingredients, students’ stand a good chance to compete.

Data analytics is all about interpretation and some of the most obscure numbers can give you the most fascinating insights into problems, helps to predict a consumer’s decision-making power. One has to go beyond the call of duty to learn from multiple fields, with the necessary perseverance, students can crack the biggest problems at the highest levels adding to our talent prowess.

Facebook Retargeting – 5 Things you just can’t miss! [PM Video Series 03]

 

Have you wished that your social media efforts could get you more than just “Likes”? Can your Facebook campaign spike up your website sales? With more than 75% of the world on Facebook, its power to help businesses grow is unimaginable.

Ever since Facebook Ads was launched in 2007, it has been on every brand’s wish list but remains an enigma. Being one of the first Facebook Marketing Partners in Asia, we have had a chance to explore, experiment and evolve retargeting strategies that have shown tremendous results for brands. Facebook Retargeting really WORKS!

And, here are 5 things you just can’t afford to miss in your Facebook Retargeting Campaign.

Segmenting of the Users:

  • Find common behaviour traits like repeat visits, products visited, time spent, etc. and segment the users showing common behaviour
  • Use different messaging across segments and align it with the behaviour.
  • Use different frequency caps for each segment

Make use of Facebook’s own data which is unique.

It captures interest and likes of people and makes you choose a more relevant audience for your campaign:

  • Add social context in terms of friends who like your page
  • Overlay your retargeting campaigns with more granular data like demographic, interest signals

Dynamic Creative Optimization- Personalise your ads for every user, in real-time.

  • Customise your messages, Keep the text short and simple
  • Use Images of the highest quality
  • Don’t forget to include call to Action for News Feed ads
  • Do Frequent Banner Refresh
  • And lastly, Use A/B testing

Have Customized Landing Pages for people who click on your ads:

  • Of course, It should be relevant to the ad, like product page for product ad, form filling page for lead generation and so on.
  • And A/B test with conversion rates to see what works best

Finally, the one thing marketers worry about when it comes to Facebook Marketing. Performance Measurement:

  • Measure performance, not in isolation, but with all channels. This is one common mistake marketers make.
  • Use standard tools like GA, Omniture for website and Apsalar like tools for Apps
  • Make a note of the reach, frequency and CPA- gives you a picture of how effective the campaign was.
  • For cross-device reporting, rely on Facebook reports

Facebook retargeting is not rocket science! It is built around people, and so is your business. Then why not your marketing? Think about it.

Vizury Whitepaper: Industry-Driven Retargeting

Online consumer behavior is significantly different for every industry. The users’ demographics and browsing / buying behavior differs based on the industry / product / category of interest. Vizury research suggests significant differences in the way people convert from retargeting ads e.g. a typical airline ticket buyer who buys after clicking a retargeting ad has on an average made 8 page visits in 3 sessions whereas an average apparel customer scores as many as 34 page visits in 4 sessions.
Another significant such industry specific trend suggests that hotels experience 90% of their retargeting conversions within 9 days whereas it takes airlines 27 days and cosmetics retailers 62 days to do the same! Our infographic based on this research is available below.

Based on this research, Vizury’s whitepaper “Industry Driven Retargeting: the benefits of a domain-driven retargeting strategy” recommends several industry-specific features that advertisers should build into their retargeting campaigns. Do download and read it. And if you have comments or questions, please email me at parthsm AT vizury DOT com.

Click on the infographic below to see a full-sized version.

Vizury Infographic Industry driven retargeting

5 Big Data Costs You Simply Can’t Afford to Ignore!

 

Everybody thinks that big data is a magical pill that you gulp, and voila, everything falls into place. One of the most common traps is to think of cost only at the level of online or front end systems. A common discussion would read something like this – “A typical $500/month machine easily handles two thousand Queries Per Second (QPS) and so if I need 10 thousand QPS more, I would need 2500$/month as an infrastructure cost”.

What this argument misses out is a lot of downstream aspects which take up bulk of the cost. We can broadly classify the entire infrastructure into five components and then try and assess them for costs.

Entry-point frontend systems

Note that these are not ‘front-end’ as in UI/UX systems, but the main entry points of your platform for the external world. Typical examples would be ad servers, beacon listeners, RTB bidders, B2C websites etc. These would invariably be core business systems that need to run 24×7. Any downtime there not only affects your revenue, but importantly, has a negative brand effect.

Naturally you plan as much buffer and redundancy here as you can afford. Surprisingly, the cost-calculations here are not hard unless your traffic fluctuates unpredictably. Its simple math like – one machine handles X QPS, let us provision for Y QPS factoring in enough buffer. Also, with techniques like auto-scaling on the cloud, you can be as close to real traffic as you want, without over-provisioning in a big way.

Storage accessed by online systems

This could typically be in-memory caching systems like or persistent stores like RDBMS or NoSQL. Tmemcache, redis his depends a lot on how you architect your systems and how much caching makes sense. A typical low-latency system like an ad server or bid server would need a lot of cache since it cannot afford to hit disks at runtime. But memory is generally super-expensive and so the architecture has to be judicious in its usage.

From a cost perspective, this component costing can be quite tricky. The thumb rule is that RAM is 10X more expensive than SSD which is 10X more expensive than HDD and performance-wise RAM is 1000x of SSD which is around 100X of HDD. With such a wide variety at your disposal, you can architect the systems creatively to have the right latency and cost. Another typical mistake done is when the frontend system scales, this caching/persistent store may not scale and so it is important to have upper limits for this benchmarked and provisioned for proper redundancy. Auto-scaling this storage system is not as easy as the stateless frontend systems.

Analytics – the meat in the sandwich

Big data in raw form is just loud noise. If you have terabytes of raw data, it’s pretty useless for the business unless a user can make sense of it. Typically, analytics involves two aspects. One is the well-defined OLAP-like reports which can be easily and quickly consumed by clients and business-users alike. Second is the adhoc queries that analysts can fire when they are trying to make business decisions. This does however, complicate the infrastructure since adhoc queries can range from a simple “count(*)” on a small table to joining multiple terabytes tables. Having a good analytics platform is a long journey. While latest technologies like hadoop, hive, spark are enabling this, none of them will be drop-in solutions which will work out of box. You will need to invest in them, tune them to your needs and strive for a balance between costs and ease-of-data-access. There is never an easy answer there!

Geographic distribution of infrastructure

It‘s not uncommon for infrastructure to be spread out around the globe. While most dedicated hosting providers make it easy, you need to be clear on the maintenance and costs involved. The first part is the bandwidth cost which is an often-understated component. Similar to RAM, one needs to be judicious with bandwidth; since it is not only expensive, but the capacities can get inconsistent. So the more data you start propagating around the world, the more uncertainty you are adding to the systems, since the bandwidth speeds are not the most reliable.

Again, when data starts scaling, a good engineering team should be able to architect a solution that will not be super-expensive and be as predictable as possible. Another angle one needs to pay heed to is how the geographical distance impacts the storage accessed by online systems. Not all systems around the world need access to data, so application and domain knowledge, backed by good architecture, will help optimize this spread.

Storage of raw data

This is generally different from storage directly accessed by online systems. Typically, this would be network storage, HDFS, S3 or the like. It’s a lot of unstructured data in raw form. While these costs are reducing by the day (S3 is hardly 0.03$/GB-month and they keep reducing it every now and then), note that the data collected is also exploding every day. Therefore, one needs to strike a balance between costs and utility of data. The next time you consider the cost of data, please factor in the entire ecosystem – one that can bring different disparate nodes into a unified umbrella, and yet justify the investments.

Originally posted on PCQuest

 

Vizury placed sixth on Deloitte Fast 500 Companies list for APAC

After being named the 2nd fastest growing technology company on the Deloitte Technology Fast 50 India 2013, Vizury has been named sixth on the APAC Fast 500 list as well. The Fast 500 list for APAC 2013 is organized by Deloitte Touche Tohmatsu Limited (DTTL) Global Technology, Media & Telecommunications (TMT) Industry Group, a ranking of the 500 fastest growing technology companies in the Asia Pacific region. Rankings are based on percentage revenue growth over three years.

The excitement on being the fastest growing ad tech company in APAC as well as in India was palpable as several employees shared the news about the recognition through social media. “We are the top ranked internet as well as ad tech company in the list for APAC. This is a considerable achievement and a testament to our leadership position in APAC when it comes to display advertising and retargeting,” says Vizury CEO, Chetan Kulkarni.

His co-founder and COO,Gourav Chindlur adds, “we have to thank our customers, partners, investors, and our employees for this spate of unabated growth. Our strong technical and data expertise in the ad tech industry has helped us gain considerable new business as well as customer loyalty. We will continue to grow rapidly in APAC as well as in other emerging geos like Middle East, North Africa and Latin America. We will be back on the top of next year’s list as well.”

About Vizury

We love the internet. Next to the wheel, it is considered one of the greatest human inventions that helps bring people closer. At Vizury, we help bring brands closer to online markets and capture not just eyeballs, but also their attention and imagination. Our display personalization solutions allow brands to optimize their targeting and retargeting campaigns and spend. Established in 2008, we work with some of the best known brands in 27 countries across industries such as travel & hospitality, ecommerce, healthcare, automobile, and classifieds. With presence in Bangalore, Beijing, Dubai, Jakarta, Sao Paolo, Singapore, Sydney, and Taiwan, we are one of the most widely networked Indian startups. Say hello to us at marketing@vizury.com.

About Deloitte

“Deloitte” is the brand under which tens of thousands of dedicated professionals in independent firms throughout the world collaborate to provide audit, consulting, financial advisory, risk management, and tax services to selected clients. These firms are members of Deloitte Touche Tohmatsu Limited (DTTL), a UK private company limited by guarantee. Each member firm provides services in a particular geographic area and is subject to the laws and professional regulations of the particular country or countries in which it operates. DTTL does not itself provide services to clients. DTTL and each DTTL member firm are separate and distinct legal entities, which cannot obligate each other. DTTL and each DTTL member firm are liable only for their own acts or omissions and not those of each other. Each DTTL member firm is structured differently in accordance with national laws, regulations, customary practice, and other factors, and may secure the provision of professional services in its territory through subsidiaries, affiliates, and/or other entities.

Vizury to strengthen global business with USD 9 Million growth funding

Bengaluru, October 22, 2012: Vizury Interactive, a leading digital marketing technology company, today announced that it has raised close to USD 9 Million in its Series B round of funding. The funding was led by Nokia Growth Partners (NGP) with participation from existing investors Ojas Ventures and Inventus Capital Partners. Existing angel investors who led the seed round of funding also continue to remain invested. The funds will be used to strengthen the company’s presence across Asia, Australia and South America, step up R&D efforts and drive product innovations.

Vizury’s premium retargeting solution Visitor Relationship Management (VRM) enables eCommerce and online travel companies to harness the full value of their digital data in a structured, multi-phase manner and profit from it. VRM is already widely used by global industry leaders such as Webjet, Virgin Airlines, Expedia, Zuji, Netshoes, Viajanet, Zozo, Kokuken, Ctrip, Yintai, Jabong, Jet Airways
and MakeMyTrip.

“We see this funding as an endorsement of our approach to strategically partner with our customers and help them drive revenue using a combination of cutting edge technology and enterprise class service” said Chetan Kulkarni, co-founder and CEO of Vizury. “The capital infusion will allow us to accelerate our investments in R&D, bring new products to market and establish ourselves as clear leaders in the space. Having NGP, Inventus and Ojas in our corner will be invaluable as we make the next big leap”, he added.

Built on a powerful digital advertising platform, VRM helps analyse massive volumes of digital data, draw meaningful insights and use it to deliver highly pertinent ads on the fly. Additionally, the fully managed VRM solution allows Vizury customers the flexibility to make the best product or service recommendations that match individual consumer needs by aligning VRM campaigns with their own programs. This has a dramatic, positive impact on conversions. With VRM, Vizury has witnessed a remarkable 450% growth over the past year. Vizury’s media partners include some of the world’s largest ad exchanges, ad networks, DSPs, SSPs and technology driven marketplaces. John Gardner, NGP managing partner in the US, said, “Vizury’s industry-leading behavioural ad targeting products and intense customer focus position the company well for expansion across a number of regions. With this investment, we reinforce NGP’s commitment to investing in the growth of the global advertising technology space”. Anupam Rastogi, principal at NGP Advisors India, who has joined Vizury’s Board, said, “We are pleased with the strong product, technology and analytic capabilities that Vizury has built. We believe Vizury’s technology has the potential to optimize customer acquisition and digital marketing for a range of clients. We look forward to working with Vizury and supporting the team in its global growth”.

About Vizury

We love the internet. Next to the wheel, it is considered one of the greatest human inventions that helps bring people closer. At Vizury, we help bring brands closer to online markets and capture not just eyeballs, but also their attention and imagination. Our display personalization solutions allow brands to optimize their targeting and retargeting campaigns and spend. Established in 2008, we work with some of the best known brands in 27 countries across industries such as travel & hospitality, ecommerce, healthcare, automobile, and classifieds. With presence in Bangalore, Beijing, Dubai, Jakarta, Sao Paolo, Singapore, Sydney, and Taiwan, we are one of the most widely networked Indian startups. Say hello to us at marketing@vizury.com.

Implementing Dynamic Thresholds Using Bischeck

When you work with a tens of publishers in this industry, there are multiple metrics to be tracked per ad-exchange such as QPS, bid rate, average bid value, hit rate, etc. and these need to be tracked across all of them. What makes this challenging is that these metrics are highly dynamic, varying based on the time of the day and the day of the week. So, how did we handle these widely dynamic metrics?

Here’s an implementation guide to help you out with that- Say Hello to Bischeck!

What is Bischeck?

Bischeck is a Nagios plugin that gives us the flexibility of using dynamic thresholds based on historical data for sending alerts. So, let us consider this with an example. One of the ad-exchange metrics that we constantly monitor is publisher QPS. This varies widely across publishers and also based on time of the day and day of the week. For example, Baidu QPS varies from 7000 during non-peak hours to 70000 during peak hours while FBX qps varies from 2000 during non-peak hours to 5000 during peak hours.

FBX_qps

 

 

 

 

 

 

 

Baidu_qps

 

 

 

 

 

 

 

So, for Baidu, while a QPS of 6000 would seem normal during non-peak hours, it would indicate a problem during peak hours. And this same 6000 could indicate an unexpected surge in FBX bid requests during non-peak hours but is something expected during peak hours. While we can use Nagios time based thresholds, scaling it across multiple publishers and multiple business metrics, involves a lot of maintenance overhead. And this is where Bischeck proved to be a boon. I will be walking through with an example on how to setup bischeck for monitoring metrics in a Ubuntu box.

Pre-requisites

1. Bischeck needs java and it uses redis as backend for storage.

$ apt-get install default-jdk redis-server

2. Currently, bischeck uses Nagios for alerting. Future versions make bischeck a standalone service. For the current version, though we will need to integrate with Nagios. Also, as it uses passive checks, we install nsca as well and configure nagios for checking external commands.

$ apt-get install nagios3 nsca
$ sed -i 's/check_external_commands=0/check_external_commands=1/' /etc/nagios3/nagios.cfg
$ service nagios3 restart
$ service nsca restart
$ dpkg-reconfigure nsca # See /usr/share/doc/nsca/README.Debian

Installation and Basic Setup

Installation of Bischeck is pretty straight forward. The latest version as of writing this article is 1.1.1. A newer version 2.0.0 with many more features [1] in the pipeline. You will have to be logged in as root.

$ wget http://gforge.ingby.com/gf/download/frsrelease/132/505/bischeck-1.1.1.tar.gz
$ tar -xvzf bischeck-1.1.1.tar.gz
$ cd bischeck-1.1.1
$ chmod 755 install
$ ./install -u # Get usage
$ ./install -I /opt/bischeck

In bischeck init script, it does “su nagios” to retrieve certain configuration parameters as user nagios. However, if nagios shell is not configured to a real shell in /etc/passwd, su will not work. So, ensure that nagios shell is set to proper shell such as /bin/bash before starting bischeck. Alternatively, if modifying nagios user shell is seen as a security concern, you can pass the option “–shell /bin/bash” to su command wherever it is being used in /etc/init.d/bischeckd (haven’t tried this though)

$ usermod --shell /bin/bash nagios

Also, bischeck submits the results to nsca. The default installation of nsca does not have any password. So, we need to remove it in bischeck config file.

$ sed -i 's/password<\/value>/password<\/value>/' /opt/bischeck/etc/services.xml/etc/servers.xml

Double check /opt/bischeck/etc/servers.xml to ensure the password is indeed removed. Finally, restart bischeck.

$ service bischeckd start
$ tail -f /var/tmp/bischeck.log

More details can be found in the Official Installation and Administration Guide [2].

Nagios Configuration

Let us create a nagios host and service to monitor. Create a file /etc/nagios3/conf.d/publisher-metrices.cfg with the following content:

# Dummy command used to alert if passive checks are not recieving data
# for specific period of time
define command {
        command_name    check_bischeck
        command_line    /usr/lib/nagios/plugins/check_dummy 1 "Results were not reported"
}

# "FBX" is not a physical host but actually one of our publishers (ad-exchange).
# Business metrics may not actually be associated with a physical machine.
# But we would still want to group together different business metrices based
# on some criteria. In our case we have multiple publishers, and each publisher
# has certain metrices associated with it (qps, error_rate, etc.). So, we have
# made the publisher name as virtual host
define host {
        use                     generic-host
        host_name               FBX
        active_checks_enabled   0
        }

# Bischeck works with passive checks. We have also added freshness check (See
# http://nagios.sourceforge.net/docs/3_0/freshness.html for more info)
define service {
        host_name               FBX
        service_description     QPS
        use                     generic-service
        active_checks_enabled   0
        passive_checks_enabled  1
        check_freshness         1
        freshness_threshold     900
        check_command           check_bischeck
}

After configuring nagios, restart it for changes to take effect.

$ service nagios3 restart
$ tail -f /var/log/nagios3/nagios.log

You can visit http://<server_ip>/nagios3/ to check that “FBX” host and “QPS” service is added. The service state will turn to warning after a while as it has not yet received any metrics.

Configuring Bischeck

First, let us create a script /usr/local/bin/FBX_qps.sh which generates dummy QPS numbers based on time of day. This will be used by bischeck.

# Get current minute between 0 to 1399
current_minute=$(( ($(date +%s) % 86400)/60 ))

# multiplier starts at 720 during 00:00 UTC, reaches zero during noon and gradually increase to 720 during midnight
multiplier=$(test $current_minute -le 720 &amp;&amp; echo $((720 - $current_minute)) || echo $(($current_minute - 720)))

# qps varies between 5000 and 55000 depending on time of day
qps=$((5000 + (50000 * $multiplier / 720)))
echo "OK | qps=$qps;0.0;0.0;;"

Make it executable and test it.

$ chmod +x /usr/local/bin/FBX_qps.sh
$ /usr/local/bin/FBX_qps.sh
OK | qps=20972;0.0;0.0;;

Next, we need to configure bischeck services. Create the file /opt/bischeck/etc/bischeck.xml with the following content (backup the existing bischeck.xml before overwriting it):

&lt;?xml version='1.0' encoding='UTF-8'?&gt;
&lt;bischeck&gt;

  &lt;host&gt;
    &lt;name&gt;FBX&lt;/name&gt;
    &lt;service&gt;&lt;template&gt;qpstemplate&lt;/template&gt;&lt;/service&gt;
  &lt;/host&gt;

  &lt;servicetemplate templatename="qpstemplate"&gt;
    &lt;name&gt;QPS&lt;/name&gt;
    &lt;schedule&gt;120S&lt;/schedule&gt;
    &lt;url&gt;shell://localhost&lt;/url&gt;

    &lt;serviceitem&gt;
      &lt;template&gt;publishermetrictemplate&lt;/template&gt;
      &lt;serviceitemoverride&gt;
        &lt;name&gt;qps&lt;/name&gt;
      &lt;/serviceitemoverride&gt;
    &lt;/serviceitem&gt;
  &lt;/servicetemplate&gt;

  &lt;serviceitemtemplate templatename="publishermetrictemplate"&gt;
    &lt;name&gt;$$SERVICEITEMNAME$$&lt;/name&gt;
    &lt;execstatement&gt;{"check":"/usr/local/bin/$$HOSTNAME$$_$$SERVICEITEMNAME$$.sh","label":"$$SERVICEITEMNAME$$"}&lt;/execstatement&gt;
    &lt;thresholdclass&gt;Twenty4HourThreshold&lt;/thresholdclass&gt;
    &lt;serviceitemclass&gt;CheckCommandServiceItem&lt;/serviceitemclass&gt;

    &lt;cache&gt;
      &lt;aggregate&gt;
        &lt;method&gt;avg&lt;/method&gt;
        &lt;useweekend&gt;true&lt;/useweekend&gt;
        &lt;retention&gt;
          &lt;period&gt;H&lt;/period&gt;
          &lt;offset&gt;720&lt;/offset&gt;
        &lt;/retention&gt;
        &lt;retention&gt;
          &lt;period&gt;D&lt;/period&gt;
          &lt;offset&gt;30&lt;/offset&gt;
        &lt;/retention&gt;
      &lt;/aggregate&gt;

      &lt;purge&gt;
        &lt;offset&gt;30&lt;/offset&gt;
        &lt;period&gt;D&lt;/period&gt;
      &lt;/purge&gt;
      &lt;!-- Max count = 30 days * 24 hours per day * 30 items per hour --&gt;
      &lt;!--
      &lt;purge&gt;
        &lt;maxcount&gt;21600&lt;/maxcount&gt;
      &lt;/purge&gt;
      --&gt;
    &lt;/cache&gt;
  &lt;/serviceitemtemplate&gt;

&lt;/bischeck&gt;

Set the appropriate thresholds by configuring /opt/bischeck/etc/24thresholds.xml. Here is a sample configuration (be sure to backup 24thresholds.xml before overwriting it):

&lt;?xml version="1.0" encoding="UTF-8" standalone="yes"?&gt;
&lt;twenty4threshold&gt;
    &lt;!-- QPS --&gt;
    &lt;servicedefgroup&gt;
        &lt;member&gt;
            &lt;hostname&gt;FBX&lt;/hostname&gt;
            &lt;servicename&gt;QPS&lt;/servicename&gt;
            &lt;serviceitemname&gt;qps&lt;/serviceitemname&gt;
        &lt;/member&gt;

        &lt;template&gt;response-qps&lt;/template&gt;
    &lt;/servicedefgroup&gt;

    &lt;servicedeftemplate templatename="response-qps"&gt;
         &lt;period&gt;
             &lt;calcmethod&gt;&amp;gt;&lt;/calcmethod&gt;
             &lt;warning&gt;20&lt;/warning&gt;
             &lt;critical&gt;40&lt;/critical&gt;
             &lt;hoursIDREF&gt;10&lt;/hoursIDREF&gt;
         &lt;/period&gt;
    &lt;/servicedeftemplate&gt;

    &lt;hours hoursID="10"&gt;
        &lt;hourinterval&gt;
            &lt;from&gt;00:00&lt;/from&gt;
            &lt;to&gt;23:00&lt;/to&gt;
            &lt;!-- See http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_configuration_guide.html#toc-Chapter-4 --&gt;
            &lt;threshold&gt;avg($$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-24H],$$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-96H],$$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-168H],$$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-336H],$$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-504H],$$HOSTNAME$$-$$SERVICENAME$$-$$SERVICEITEMNAME$$[-672H])&lt;/threshold&gt;
        &lt;/hourinterval&gt;
    &lt;/hours&gt;
&lt;/twenty4threshold&gt;

Finally, restart bischeck.

$ service bischeckd restart
$ tail -f /var/tmp/bischeck.log

If everything is fine, you should start seeing the QPS values in Nagios web console.

Bischeck also provides integration with pnp4nagios [5]. Configuring this is beyond the scope of this article though.

Bischeck Command Line Utilites

Bischeck provides a couple of very useful command line tools for debugging [6].

The first is for threshold testing.

$ /opt/bischeck/bin/bischeck threshold.Twenty4HourThreshold -h FBX -s QPS -i qps -d 20150513 -H 08 -M 30 -m 10000

Bischeck provides a command line tool to check if values are getting populated properly and also for evaluating expressions:

$ /opt/bischeck/bin/bischeck cli.CacheCli
cacehcli&gt; avg(FBX-QPS-qps[0:4])
[1/1/2 ms] avg(10972,11111,11250,11388,11527) = 11249.6

Closing Comments

As it can be seen, bischeck is a pretty powerful plugin and provides a lot of functionalities beyond just dynamic thresholds. Moreover, they provide excellent support and the developers have been very responsive in incorporating feedback as well as fixing bugs. For any organization with highly dynamic business metrics, bischeck is definitely a must have!

References

[1] http://www.bischeck.org/?p=946
[2] http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html
[3] http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_configuration_guide.html
[4] http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-4.4
[5] http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-3.3
[6] http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-4.3

Looking for an all in one 360 solution to grow marketing ROI?

Leverage Engage360 to drive meaningful relationships with your customers.

Plugin Support By WordPress Premium Plugins