The changing nature of application development necessitates new methods for ensuring updates and patches are deployed in near-real time via automated testing and validation. The challenge is to deploy apps in a way that doesn’t hinder users’ access to your organization’s vital applications. To do so, you have to think small, and you have to be ready to respond to feedback as quickly as you receive it.

It’s almost certain that whatever software components your new application needs have already been written. “Application development” is now a matter of putting pre-existing software pieces together in an intelligent, problem-solving, revenue-generating way, with little or no original coding required. Then all you need is the most efficient combination of compute cycles, storage, network bandwidth, and database elements to support your apps.

What matters as much as development these days is continuous, almost-instantaneous updates, so you’re in a near-constant state of deployment. The goal is to update in the background, and in a way that minimizes interference with users.

1. Deployments: Small and focused
2. Testing: Fast and well-defined

The foundation of modern app development is API- and service-based access to functionality: Code and configuration repositories, source code, and configuration versioning and controls are required for rapid, focused, compartmentalized deployment. One key to the success of your app-development project is to involve the “Ops” side of DevOps from the get-go. This way, the operations staff can help define the supportability, testing, management, and security of the services comprising your applications.

Updates will often add new fields, modify forms and reports, customize or enhance dashboards, even add new use cases to expand the potential user base for the application. All of these changes make the app more complex. They also have the potential to conflict with other infrastructure components, and they may hinder performance and usability.

You can’t always know what changes you’ll have to make to the app in the future to keep it up-to-date. You have to develop with flexibility in mind. When you take a no-code, low-code approach to app development, you’re better able to make alterations quickly and precisely, with minimal impact on users.

Continuous deployment depends on two things: Determining the lowest level of test coverage necessary for each deployment, and choosing automated testing tools that you can trust to do the job. In a November 13, 2015, post on DZone’s DevOps Zone, Moritz Plessnig identifies three principal benefits of continuous deployment:

1. Small, focused iterations are less likely to cause problems, and they’re also easier to troubleshoot.
2. Frequent updates maintain momentum and give users a sense of the app progressing.
3. Users get faster access to new features and are able to provide feedback quicker.

Continuous deployment moves changes through the deployment cycle automatically. Source: Yaniv Yehuda, via InfoQ 

Plessnig considers three types of deployment tests to be indispensable:

1. The smoke test simply confirms that the system functions as it was designed to function. You want to make sure the various components work as expected in the production environment, not just in a simulated testbed.
2. Happy path story testing ensures that the sequence of actions users typically step through to complete a task operate as expected. Examples are the sign-in process, order entry, and product purchase. You have to test the “happy path” for each of these “user stories,” according to Plessnig. As with smoke tests, the happy path test must use production code and run in a production environment.
3. Integration testing verifies the application’s boundaries. It can take the place of unit testing, generative testing, mutation testing, and other traditional developer-focused tests. For example, classes and modules may run without a hitch in an isolated test environment but clash when they’re connected to a database, file system, or other external component.

One of the most popular ways to keep tabs on the configuration of your Linux servers is Puppet, which is available in both open-source and commercial versions. As reported in a November 25, 2015, article in Linux Journal, after you define the state of your IT infrastructure, Puppet enforces that state across OSes; physical, virtual, and cloud environments; network and storage devices; and applications. This includes initial code development, testing, production release, and updates; as well as provisioning of physical and virtual machines, orchestration, and reporting.

In the Puppet workflow, the agent collects information about the host and sends it to the server; the parser compiles an implementation for the host and returns it to the agent; and then the agent applies the configuration locally. Source: Luke Kanies, via Aosabook.org

The new Puppet Application Orchestration App is intended to serve as the single method for updating all of your application’s deployment tiers: from the application layer through all the infrastructure layers it supports. This lets you model distributed applications as Puppet code to take advantage of Puppet’s declarative model-based design. More than 3,500 Puppet modules are available for assembly into full models of distributed applications.

Morpheus helps customers of all sizes reduce application deployment time by more than 2x. To see how Morpheus can help your business, click here to sign up for a free demo. 

It's getting easier and easier for criminals to infiltrate your company’s network and help themselves to your financial and other sensitive information, and that of your customers. There's a ready market for stolen certificates that make malware look legitimate to antivirus software and other security systems.

The crooks even place orders for stolen account information: One person is shopping for purloined Xbox, GameStop, iTunes, and Target accounts; another is interested only in accounts belonging to Canadian financial institutions. Each stolen record costs from $4 to $10, on average, and customers must buy at least $100 worth of these hijacked accounts. Many of the transactions specify rubles (hint, hint).

Loucif Kharouni, Senior Threat Researcher for security service Damballa, writes in a September 21, 2015, post that the cybercrime economy is thriving on the so-called Darknet, or Dark Web. Criminals now offer cybercrime-as-a-service, allowing anyone with an evil inclination to order up a malware attack, made to order -- no tech experience required.

Sadly, thieves aren't the only criminals profiting from the Darknet. Human traffickers, child pornographers, even murderers are taking advantage of the Internet to commit their heinous crimes, as Dark Reading's Sara Peters reports in a September 16, 2015, article.

Peters cites a report by security firm Bat Blue Networks that claims there are between 200,000 and 400,000 sites on the Darknet. In addition to drug sales and other criminal activities, the sites are home to political dissidents, whistleblowers, and extremists of every description. It's difficult to identify the servers hosting the sites because they are shrouded by virtual private networks and other forms of encryption, according to Bat Blue's researchers.

Most people access the sites using The Onion Router (Tor) anonymizing network. That makes it nearly impossible for law enforcement to identify the criminals operating on the networks, let alone capture and prosecute them. In fact, Bat Blue claims "nation-states" are abetting the criminals, whether knowingly or unknowingly.

The Darknet is populated by everyone from public officials to religious extremists, for as wide a range of purposes. Source: Bat Blue Networks

While hundreds of thousands of sites comprise the Darknet, you won’t find them using the web’s Domain Name System. Instead, the sites communicate by delivering an anonymous service, called a “hidden service,” via updates to the Tor network. Rather than getting a domain from a registrar, the sites authenticate each other by using self-generated public/private key pair addresses.

The public key generates a 16-character hash that ends in .onion to serve as the address that accesses the hidden service. When the connection is established, keys are exchanged to create an encrypted communication channel. In a typical scenario, the user installs a Tor client and web server on a laptop, takes the laptop to a public WiFi access point (avoiding the cameras that are prevalent at many such locations), and uses that connection to register with the Tor network.

The Tor Project explains the six-step procedure for using a hidden service to link anonymously and securely via the Tor network:

1. Party A builds circuits to select introduction points on the Tor network.
2. A hidden service descriptor containing the public key and summaries of each introduction point, and signed by the private key, is uploaded to a distributed hash table on the network.
3. Party B finds the hidden service’s .onion address and downloads the descriptor from the distributed hash table to establish the protected connection to it.
4. Party B creates an “introduce” message encrypted to the hidden service's public key; the message includes the address of the rendezvous point and the one-time secret. The message is sent to one of the introduction points for delivery to the hidden service. (This step is shown in the image below.)
5. The hidden service decrypts the message, finds the rendezvous address and one-time secret, creates a circuit to the rendezvous point, and sends a rendezvous message that contains another one-time secret.
6. The rendezvous point notifies Party B that the connection has been established, and then Party B and the hidden service pass protected messages back and forth.

In the fourth of the six steps required to establish a protected connection to the Tor network, Party B (Ann) sends an “introduce” message to one of the hidden service’s introduction points created by Party A (Bob). Source: Tor Project

Criminals cannot be allowed to operate unfettered in the dark shadows of the Internet. But you can’t arrest what you can’t spot. That’s why the first step in combatting Darknet crime is to shine a light on it. That’s one of the primary goals of the U.S. Defense Research Projects Agency’s Memex program, which Mark Stockley describes in a February 16, 2015, post on the Sophos Naked Security site.

Memex is intended to support domain-specific searches, as opposed to the broad, general scope of commercial search engines such as Google and Bing. Initially, it targets human trafficking and slavery, but its potential uses extend to the business realm, as Computerworld’s Katherine Noyes reports in a February 13, 2015, article.

For example, a company could use Memex to spot fraud attempts and vet potential partners. However, the ability to search for information that isn’t indexed by Google and other commercial search engines presents companies with a tremendous competitive advantage, according to analysts. After all, knowledge is power, and not just for the crooks running amok on the Darknet.

Find out how a CLI can help your company with app development and deployment 

A CLI can help developers and administrators get apps deployed quickly, in addition to the other uses of CLIs, such as automating tasks for an operating system or running helpful commands that can run a ping, get information on files, or execute file system commands.

CLI stands for command-line interface, though you might also hear of this under a different name or acronym, such as character user interface (CUI) or command-line user interface. A CLI allows a user to commands to a program using written text rather than a GUI (Graphical User Interface). For many years, CLIs were the main form of interacting with computers. The introduction of GUI operating systems such as Windows and Macintosh eventually made computer operation easier for beginners, but CLIs have remained as an alternate means of interaction, often employed by more advanced users such as administrators or developers.

Source: HubPages 

For administrators and developers, a CLI can be a handy tool for interacting with a particular piece of software. Whether it is an operating system, server, or another system, a CLI can provide you with the ability to provide additional parameters to the commands that you need to run. This sort of flexibility helps when you need to configure a command to run a particular way. For example, getting a directory listing in a CLI for Windows is done by typing “dir” and pressing “Enter”. This works well for short listings, but if that particular directory has a large number of files, you may not be able to see what you needed on the screen. To help with this, you can add the “/p” parameter, e.g., “dir /p”. This time, the CLI will display the number of files that will fit on a single page and allow you to continue as needed until you see what you need.

As an administrator or developer, you may be considering a SaaS package, and having a CLI to work with would be helpful. Morpheus is a service that offers exactly what you need: SaaS with a CLI for you to help your productivity. With the Morpheus CLI, you can quickly provision instances by typing in the necessary commands. For example, the interaction below will create a Node.js instance on Morpheus with ease: 

 $ morpheus instances add "My node app" node

Configurations:

  1) Single Node (node-4.0.0-single)

Selection: 1

Select a Plan:

  1) Memory: 128MB Storage: 1GB

  2) Memory: 256MB Storage: 3GB

  3) Memory: 512MB Storage: 5GB

  4) Memory: 1GB Storage: 10GB

  5) Memory: 2GB Storage: 20GB

  6) Memory: 4GB Storage: 40GB

  7) Memory: 8GB Storage: 80GB

  8) Memory: 16GB Storage: 160GB

Selection: 1

With Morpheus, you can provision apps and databases in real-time to public, private, and hybrid clouds and spin up databases, apps, environments and more with a few simple clicks. You can use the monitoring service to keep track of overall app uptime and response time, while also tracking the vital statistics for each individual piece of your app. With all of these features, why not give Morpheus a try today? Click here to sign up for a demo now. 

Example of a MongoDB query. Source: MongoDB.

MongoDB was originally developed by MongoDB, Inc., which at the time (2007) was named 10gen. The company was founded by former DoubleClick founders and engineers, specifically Dwight Merriman, Kevin P. Ryan, and Eliot Horowitz.

At first, 10gen wanted to build an open-source platform as a service. The company wanted all of the components of its software to be completely open-source, but could not find a database that met their needs and provided the type of scalability needed for the applications they were building.

The platform 10gen was working on was named Babble and was going to be similar to the Google App Engine. As it turned out, there wasn't a big market for Babble, but both users and non-users of Babble agreed that the database 10gen had created to accompany the platform was excellent and would be happy to use it on its own.

While originally simply dubbed "p", the database was officially named MongoDB, with "Mongo" being short for the word humongous. Given the input 10gen had received about MongoDB, the company decided it would indeed be best to scrap the Babble project and release MongoDB on its own as an open-source database platform in 2009.

By 2012, 10gen had been named number nine on "The Next Big Thing 2012" published by the Wall Street Journal and had 6 offices located in various parts of the world. In 2013, 10gen renamed itself to MongoDB, Inc., wanting to make the strong association with its popular primary product.

As time went on, MongoDB moved up the ranks to become the most popular type of database for document stores, and the fourth most popular database system overall. It is used by other highly successful companies like eBay, Abobe, LinkedIn, Foursquare, McAfee, Shutterfly, and others.

It is also used by software developers as part of the MEAN stack, which includes MongoDB (database), Express (web app framework), AngularJS (MVC JavaScript front-end framework) and NodeJS (platform for server-side apps). Part of the popularity of this stack is that JavaScript and/or JSON/BSON notation can be used across all members of the stack, allowing developers to easily move through and develop within each piece of the stack.

The MEAN stack. Source: modernweb.

All in all, MongoDB can be an excellent choice for a database for your applications, especially if you deal with large amounts of data that will continually expand over time!

To see how Morpheus can help you get more out of your MongoDB sign up for a demo today!

By Nermin Hajdarbegovic, Technical Editor at Toptal

At a glance, the hosting industry may not appear exciting, but it's grunts in data centres the world over that keep our industry going. They are, quite literally, the backbone of the Internet, and as such they make everything possible: from e-commerce sites, to smart mobile apps for our latest toys. The heavy lifting is done in boring data centres, not on our flashy smartphones and wafer thin notebooks. 

Whether you’re creating a virtual storefront, deploying an app, or simply doing some third-party testing and development, chances are you need some server muscle. The good news is that there is a lot to choose from. The hosting industry may not be loud or exciting, but it never sleeps; it’s a dog eat dog world, with cutthroat pricing, a lot of innovation behind the scenes, and cyclical hardware updates. Cloud, IaaS and PaaS have changed the way many developers and businesses operate, and these are relatively recent innovations.

In this post I will look at some hosting basics from the perspective of a freelance developer: what to choose and what to stay away from. Why did I underline freelance software engineers? Well, because many need their own dev environment, while at the same time working with various clients. Unfortunately, this also means that they usually have no say when it comes to deployment. For example, it’s the client’s decision how and where a particular web app will be hosted, and a freelancer hired on short-term basis usually has no say in the decision. This is a management issue, so I will not address it in this post other than to say that even freelancers need to be aware of options out there. Their hands may be tied, but in some cases clients will ask for their input and software engineers should help them make an informed decision. Earlier this week, we covered one way of blurring the line between development and operations: DevOps. In case you missed that post, I urge you to check it out and see why DevOps integration can have an impact on hosting as well.

Luckily, the hosting industry tries to cater to dev demand, so many of hosting companies offer plans tailored for developers. But wait, aren’t all webhosting plans just as good for developers as these “developer” plans? Is this just clever marketing and a cheap SEO trick?

So, how does one go about finding the right hosting plan? Google is the obvious place to start, so I tried searching for “hosting for developers.” By now, you can probably see where I am going with this. That particular search yielded 85 million results and enough ads to make Google shareholders pop open a bottle of champagne.

If you’re a software engineer looking for good hosting, it’s not a good idea to google for answers. Here’s why.

There is a very good reason for this, and I reached out to some hosting specialists to get a better idea of what goes on behind the scenes.

Adam Wood, Web Hosting Expert and Author of Ultimate Guide to Web Hosting explained: 

“Stay away from Googling ‘hosting for developers.’ That shows you hosts that have spent a lot of money on SEO, not a lot of energy on building an excellent platform.” 

Wood confirmed what most of us knew already: A lot of “hosting for developers” plans are marketing gimmicks. However, he stressed that they often offer perfectly fine hosting plans in their own right.

“The ‘hosting’ is real, the ‘for developers’ part is just marketing,” he added.

Although Wood works for hosting review site WhoIsHostingThis, he believes developers searching for a new host should rely on more than online searches.

Instead of resorting to Google, your best bet for finding the perfect plan for your dev needs is word of mouth and old-fashioned research:

• Check out major tech blogs from developers using the same stack as you.
• Reach out to the community and ask for advice.
• Take a closer look at hosting plans offered by your current host. Look for rapid deployment tools, integration to other developer tools, testing support and so on.
• Make sure you have clear needs and priorities; there’s no room for ambiguity.
• Base your decision on up-to-date information.

But what about the hundreds of thousands of hosting plans tailored for developers? Well, they’re really not special and in most cases you can get a similar level of service and support on a “plain Jane” hosting plan.

Is there even a need for these small and inexpensive plans? Yes, there is. Although seasoned veterans probably won’t use them, they are still a piece of the puzzle, allowing small developers, hobbyists and students to hone their skills on cheap, using shared hosting plans that cost less than a gym membership. Nobody is going to host a few local hobby sites on AWS, and kids designing their first WordPress sites won’t get a VPS. In most cases, they will use the cheapest option out there.

Cheap, shared hosting plans are the bread and butter of many hosting outfits, so you can get one from an industry leader, or a tiny, regional host. The trouble with small hosts is that most of them rely on conventional reseller hosting or re-packaging cloud hosting from AWS and other cloud giants. These plans are then marketed as shared hosting plans, VPS plans, or reseller plans.

Bottom line: If something goes wrong with your small reseller plan, who are you going to call in the middle of the night?

Small hosts are fading and this is more or less an irreversible trend. Data centres are insanely capital-intensive; they’re the Internet equivalent of power stations, they keep getting bigger and more efficient, while at the same time competing to offer lower pricing and superior service. This obviously involves a lot of investment, from huge facilities with excellent on-site security and support through air-conditioning, redundant power supply and amazingly expensive Internet infrastructure. On top of that, hosts need a steady stream of cutting edge hardware. Flagship Xeons and SAS SSDs don’t come cheap.

There is simply no room for small players in the data centre game.

Small resellers still have a role to play, usually by offering niche services or a localisation, including local support in various languages not supported by the big host. However, most of these niches and potential advantages don’t mean a whole lot for the average developer.

Less than a decade ago, the industry revolved around dedicated and shared hosting, and I don’t think I need explain what they are and how they work.

Cloud services entered the fray a few years ago, offering unprecedented reliability and scalability. The latest industry trends offer a number of exciting possibilities for developers in the form of developer-centric Platform-as-a-Service (PaaS) offerings.

PaaS is the new black for many developers. How does it compare to traditional hosting?

Most developers are already familiar with big PaaS services like Heroku, Pantheon and OpenShift. Many of these providers began life as platforms for a specific framework or application. For example, Heroku was a Ruby-on-Rails host, while Pantheon was a Drupal managed-hosting provider, which expanded to WordPress.

PaaS services can be viewed as the next logical step in the evolution of managed hosting. However, unlike managed hosting, PaaS is geared almost exclusively toward developers. This means PaaS services are tailored to meet the needs of individual developers and teams. It’s not simply about hosting; PaaS is all about integrating into a team’s preferred workflow by incorporating a number of features designed to boost productivity. PaaS providers usually offer a host of useful features:

·       Ability to work with other developer tools like GitHub.

·       Supports Continuous Integration (CI) tools like Drone.io, Jenkins, and Travis CI.

·       Allows the creation of multiple, clonable environments for development, testing, beta, and production.

·       Supports various automated testing suites.

Best of all, many PaaS providers offer free developer accounts. Heroku and Pantheon both allow developers to sample the platform, thus encouraging them to use it for projects later on. In addition, if one of these experimental projects takes off, developers are likely to remain on the platform. 

It’s clever marketing, and it’s also an offer a lot of developers can’t afford to ignore. PaaS is here to stay and if you haven’t taken the plunge yet, perhaps it is time to do a little research and see what’s out there.

Dedicated and shared hosting aren’t going anywhere. They were the mainstays of web hosting for two decades and they’re still going strong. A lot of businesses rely on dedicated servers or VPS servers for their everyday operations. Some businesses choose to use cloud or PaaS for specific tasks, alongside their existing server infrastructure.

In some situations, PaaS can prove prohibitively expensive, but powerful dedicated servers don’t come cheap, either. The good news is that PaaS can give you a good idea of the sort of resources you will need before you decide to move to a dedicated server. Further, PaaS services tend to offer better support than managed VPS servers or dedicated servers.

Of course, all this is subjective and depends on your requirements and budget.

PaaS, dedicated servers, VPS plans, or your own slice of the Cloud. What should a freelance software engineer choose?

Call me old-fashioned, but I still believe dedicated servers are the best way of hosting most stuff. However, this only applies to mature projects; development is a whole other ball game. Managed dedicated servers offer exceptional reliability and good levels of support, along with good value for money.

Properly used, dedicated servers and PaaS can speed up deployment as well, as Adam Wood explains:

“I can spin up a new Ruby-on-Rails app on Heroku in a matter of minutes. Doing the same thing on AWS takes me a half a day, and I constantly feel like I’m about to break something.”

Cloud services are inherently more efficient than dedicated hardware because you only use the resources you need at any given time. For example, if you are operating a service that gets most of its traffic during office hours (from users in the Americas), your dedicated server will be underutilised for 12 to 16 hours. Despite this obvious efficiency gap, dedicated servers can still end up cheaper than cloud solutions. In addition, customers can customise and upgrade them the way they see fit.

Cloud is catching up, but dedicated servers will still be around for years to come. They’re obviously not a good solution for individual developers, but are for a lot of businesses. VPS plans cost a lot less than dedicated servers and are easily within the reach of individual developers, even though they don’t offer the same level of freedom as dedicated servers.

The good news is that most freelance software engineers don’t need to worry about every hosting option out there. While it’s true that different clients have different ways of doing things, in most cases it’s the client’s problem rather than yours.

This does not mean that different hosting choices have no implications on freelancers; they do, but they are limited. It is always a good idea to familiarise yourself with the infrastructure before getting on board a project, but there is not much to worry about. Most new hosting services were developed to make developers’ lives easier and keep them focused on their side of the project. One of the positive side-effects on PaaS and cloud adoption is increasing standardisation; most stacks are mature and enjoy wide adoption, so there’s not a lot that can go wrong.

Besides, you can’t do anything about the client’s choice of infrastructure, for better or for worse. But what about your own server environment?

There is no one-size-fits-all solution; it all depends on your requirements, your stack, and your budget. PaaS services are gaining popularity, but they might not be a great solution for developers on a tight budget, or those who don’t need a hosting environment every day. For many freelancers and small, independent developers, VPS is still the way to go. Depending on what you do, an entry-level managed dedicated server is an option, and if you do small turnkey web projects, you may even consider some reseller packages. 

The fact that big hosting companies continue to compete for developers’ business is, ultimately, a good thing. It means they’re forced to roll out timely updates and offer better support across all hosting packages in order to remain competitive. They are not really competing with PaaS and cloud services, but they still want a slice of the pie.

Remember how PaaS providers offer developers various incentives to get on board, just so they could get their business in the long run? It could be argued that conventional hosting companies are trying to do the same by luring novice developers to their platform, hoping that they will be loyal customers and use their servers to host a couple of dozen projects a few years down the road.

Although the hosting industry may not appear as vibrant and innovative as other tech sectors, this is not entirely fair. Of course, it will always look bland and unexciting compared to some fast-paced sectors, but we’re talking about infrastructure, not some sort of get rich quick scheme.

The hosting industry is changing, and it is innovative. It just takes a bit longer to deploy new technology, that’s all. For example, a logistics company probably changes its company smartphones every year or two, but its delivery vehicles aren’t updated nearly as often, yet they’re the backbone of the business.

Let’s take a quick look at some hosting industry trends that are becoming relevant from a software development perspective:

·       Continual development and growth of Cloud and PaaS services.

·       Evolution of managed hosting into quasi-PaaS services.

·       Increasing integration with industry standard tools.

·       New hardware might make dedicated servers cheaper.

Cloud and PaaS services will continue to mature and grow. More importantly, as competition heats up, prices should come down. The possibility of integrating various development tools and features into affordable hosting plans will continue to make them attractive from a financial perspective. Moving up on the price scale, managed hosting could also evolve to encompass some features and services offered by PaaS. If you’re interested in hosting industry trends, I suggest you check out this Forbes compilation of cloud market forecasts for 2015 and beyond.

Dedicated servers will never be cheap, at least not compared to shared and VPS plans. However, they are getting cheaper, and they could get a boost in the form of frugal and inexpensive ARM hardware. ARM-based processors tend to offer superior efficiency compared to x86 processors, yet they are relatively cheap to develop and deploy. Some flagship smartphones ship with quad-core chips, based on 64-bit Cortex-A57 CPU cores, and the same cores are coming to ARM-based server processors.

As a chip geek, I could go on, but we intend to take an in-depth look at the emerging field of ARM servers in one of our upcoming blog posts, so if you’re interested, stay tuned.

This article originally appeared in Toptal link at https://www.toptal.com/it/hosting-for-freelance-developers-paas 

To try out Morpheus' leading PaaS offering sign up for a free demo here. 

Data logging is often talked about as a helpful tool that you can use when trying to maintain your various servers, databases, and other systems that go into an application. So, what is data logging and what does it do that helps you maintain your applications more easily?

Data Logging Defined Generally speaking, data logging is the recording of data over a period of time by a computer system or a special standalone device which can be tailored to a specific use case. The recorded data can then be retrieved and analyzed to help determine if things ran smoothly during the time the data was being recording, and to help identify what happened if there were any issues that would be in need or further attention. Standalone data loggers are used in many familiar environments to gather information such as weather conditions, traffic conditions, wildlife research, and many others. These devices make it possible for the recording of data to take place 24/7 and automatically, without the need for a person to be present with the data logger. 

A data logger for a weather station. Source: Wikipedia.

For instance, when performing wildlife research, it can be beneficial to have such automated logging, as wildlife may behave differently when one or more humans are present. For the purposes of application monitoring, data logging records information pertinent to the maintenance of the infrastructure that is required for an application to run.

How Data Logging Helps With App Maintenance When maintaining apps, it is always helpful to know when and where something went wrong. In many cases, such logging can help you avoid problems by alerting you that an issue may arise soon (a server beginning to respond slowly, for instance). Data logging can also help you keep track of statistics over time, such as the overall uptime, the uptime of specific servers, average response time, and other data that can help you tweak your applications for optimum uptime and performance.

If you are looking for a monitoring system with excellent data logging and analysis reports, you should give Morpheus a try. With Morpheus, data logging is automatic as you provision servers and apps. Using the available tools, you can monitor the various parts of your system to keep track of uptime, response time, and to be alerted if an issue does arise.

The Morpheus interface is clean and easy to use. Source: Morpheus.

Morpheus also allows you to provision apps in a single click and provides ease of use for developers with APIs and a CLI. In addition, backups are also automatic, and you can have redundancy as needed to avoid potentially long waits for disaster recovery to take place. Sign up for a demo and we'll let you try out Morpheus for free today. 

If you need to use pagination in your Redis app, there are a couple of strategies you can use to achieve the necessary functionality. While pagination can be challenging, a quick overview of each of these techniques should be helpful in making your job of choosing a method and implementing it a little easier. There are several strategies for pagination in Redis. Find out what they are and the pros and cons of each!

In Redis, you have a couple of options from which to choose. You can use the SSCAN command or you can use sorted sets. Each of these has their own advantages, so choose the one that works best for your application and its infrastructure.

The SSCAN command is part of a group of commands similar to the regular SCAN command. These include:

• SCAN - Used to iterate over the set of keys in the current database.
• SSCAN - Used to iterate over elements of sets.
• HSCAN - Used to iterate fields hashes and associated values.
• ZSCAN - Used to iterate elements of sorted sets and their scores.

Example of scan iteration. Source: Redis.

So, while the regular SCAN command iterates over the database keys, the SSCAN command can iterate over elements of sets. By using the returned SSCAN cursor, you could paginate over a Redis set.

The downside is that you need some way to persist the value of the cursor, and if there are concurrent users this could lead to some odd behavior, since the cursor may not be where it is expected. However, this can be useful for applications where traffic to these paginated areas may be lighter.

In Redis, sorted sets are a non-repeating collection of strings associated with a score. This score is used to order the set from the smallest to the largest score. This data type allows for fast updating by giving you easy access to elements, even if the elements are in the middle of the set.

An example of sorted set elements Source: Redis.

To paginate, you can use the ZRANGE command to select a range of elements in a sorted set based on their scores. So, you could, for example, select scores from 1-20, 21-40, and so on. By programmatically adjusting the range as the user moves through the data, you can achieve the pagination you need for your application.

Since sorted sets and ZRANGE do this task more intuitively than using a scan, it is often the preferred method of pagination, and is easier to implement with multiple users, since you can programmatically keep track of which ZRANGE each user is selecting at any given time.

In the end, you can choose which method works for your particular situation. If you have a smaller application with less traffic, a scan may work for you. If; however, you need a more robust solution for larger data sets or more highly utilized applications, it may be best to go ahead and use ZRANGE with sorted sets to achieve pagination in your application.

Morpheus helps you get more out of Redis. To find our how Morpheus can save you time, money and sanity sign up for a demo now!

At times, a web app or web site may appear to be down when the server it is on appears to be functioning properly. When this happens, it is important to know where the issue resides, as it may be easy to fix, or may require a lot of work or contacting others. One of the possibilities when a site is in this state is whether or not the DNS server is up to date and pointing others to the proper server in order to load your site or app.

DNS stands for Domain Name System. It is the tool that allows a typical URL, such as http://gomorpheus.com , to point to the server on which the actual web site or app resides. Once a computer finds the DNS information it needs for mapping a base URL to a server address, it will remember it for a period of time, until its TTL (Time To Live) has been reached.

DNS can contribute to downtime in several ways:

1. The DNS server has the wrong information stored about the server to which the domain should be pointed. For example, the server is actually at the IP address 204.268.130.100, but the DNS entry has the server at 204.268.120.100. Here, changing the entry to the proper address will fix the situation.
2. The DNS server is down. In such a case, computers that do not have the DNS information cached cannot reach the DNS server to look up the proper address. This will require getting your DNS server back up and running, or contacting the proper people to do this if it is not your server.
3. The changes haven’t propagated and updated caches yet. Since computers cache DNS information in the operating system and browser, this could be the case.

If the user is affected by number three above, there are a couple of things to try:

1. Have the user close the web browser, reopen it, and try again. Browsers have a tendency to cache DNS information, so this may solve the issue.
2. Have the user clear the DNS cache on their operating system. This can be done from a shell, for example, the commands to do this in Windows and OSX are shown below:

#Windows:

ipconfig /flushdns

#OSX:
sudo killall -HUP mDNSResponder

Examples of clearing the DNS cache

Do you want to be notified when your site or app is having issues? If you are looking for a monitoring system with excellent data logging and analysis reports, you should give Morpheus a try. With Morpheus, data logging is automatic as you provision servers and apps. Using the available tools, you can monitor the various parts of your system to keep track of uptime, response time, and to be alerted if an issue does arise.

The Morpheus interface is clean and easy to use. 

Morpheus allows you to provision apps in a single click, and provides ease of use for developers with APIs and a CLI. In addition, backups are also automatic, and you can have redundancy as needed to avoid potentially long waits for disaster recovery to take place. So, why not register an account or try out Morpheus for free today?

By Gergely Kalman, Security Specialist at Toptal

For all too many companies, it’s not until after a breach has occurred that web security becomes a priority. During my years working as an IT Security professional, I have seen time and time again how obscure the world of IT Security is to so many of my fellow programmers.

An effective approach to IT security must, by definition, be proactive and defensive. Toward that end, this post is aimed at sparking a security mindset, hopefully injecting the reader with a healthy dose of paranoia.

In particular, this guide focuses on 10 common and significant web security pitfalls to be aware of, including recommendations on how they can be avoided. The focus is on the Top 10 Web Vulnerabilities identified by the Open Web Application Security Project (OWASP), an international, non-profit organization whose goal is to improve software security across the globe.

When speaking with other programmers and IT professionals, I often encounter confusion regarding the distinction between authorization and authentication. And of course, the fact the abbreviation auth is often used for both helps aggravate this common confusion. This confusion is so common that maybe this issue should be included in this post as “Common Web Vulnerability Zero”.

So before we proceed, let’s clearly the distinction between these two terms:

• Authentication: Verifying that a person is (or at least appears to be) a specific user, since he/she has correctly provided their security credentials (password, answers to security questions, fingerprint scan, etc.).
• Authorization: Confirming that a particular user has access to a specific resource or is granted permission to perform a particular action.

Stated another way, authentication is knowing who an entity is, while authorization is knowing what a given entity can do. 

Injection flaws result from a classic failure to filter untrusted input. It can happen when you pass unfiltered data to the SQL server (SQL injection), to the browser (XSS – we’ll talk about this later), to the LDAP server (LDAP injection), or anywhere else. The problem here is that the attacker can inject commands to these entities, resulting in loss of data and hijacking clients’ browsers. 

Anything that your application receives from untrusted sources must be filtered, preferably according to a whitelist. You should almost never use a blacklist, as getting that right is very hard and usually easy to bypass. Antivirus software products typically provide stellar examples of failing blacklists. Pattern matching does not work. 

Prevention: The good news is that protecting against injection is “simply” a matter of filtering your input properly and thinking about whether an input can be trusted. But the bad news is that all input needs to be properly filtered, unless it can unquestionably be trusted (but the saying “never say never” does come to mind here).

In a system with 1,000 inputs, for example, successfully filtering 999 of them is not sufficient, as this still leaves one field that can serve as the Achilles heal to bring down your system. And you might think that putting an SQL query result into another query is a good idea, as the database is trusted, but if the perimeter is not, the input comes indirectly from guys with malintent. This is called Second Order SQL Injection in case you’re interested.

Since filtering is pretty hard to do right (like crypto), what I usually advise is to rely on your framework’s filtering functions: they are proven to work and are thoroughly scrutinized. If you do not use frameworks, you really need to think hard about whether not using them really makes sense in your environment. 99% of the time it does not.

This is a collection of multiple problems that might occur during broken authentication, but they don’t all stem from the same root cause.

Assuming that anyone still wants to roll their own authentication code in 2014 (what are you thinking??), I advise against it. It is extremely hard to get right, and there are a myriad of possible pitfalls, just to mention a few:

1. The URL might contain the session id and leak it in the referer header to someone else.
2. The passwords might not be encrypted either in storage or transit.
3. The session ids might be predictable, thus gaining access is trivial.
4. Session fixation might be possible.
5. Session hijacking might be possible, timeouts not implemented right or using HTTP (no SSL), etc…

Prevention: The most straightforward way to avoid this web security vulnerability is to use a framework. You might be able to implement this correctly, but the former is much easier. In case you do want to roll your own code, be extremely paranoid and educate yourself on what the pitfalls are. There are quite a few.

This is a fairly widespread input sanitization failure (essentially a special case of common mistake #1). An attacker gives your web application JavaScript tags on input. When this input is returned to the user unsanitized, the user’s browser will execute it. It can be as simple as crafting a link and persuading a user to click it, or it can be something much more sinister. On page load the script runs and, for example, can be used to post your cookies to the attacker.

Prevention: There’s a simple web security solution: don’t return HTML tags to the client. This has the added benefit of defending against HTML injection, a similar attack whereby the attacker injects plain HTML content (such as images or loud invisible flash players) – not high-impact but surely annoying (“please make it stop!”). Usually, the workaround is simply converting all HTML entities, so that script is returned as <script>. The other often employed method of sanitization is using regular expressions to strip away HTML tags using regular expressions on < and > , but this is dangerous as a lot of browsers will interpret severely broken HTML just fine. Better to convert all characters to their escaped counterparts. 

This is a classic case of trusting user input and paying the price in a resulting security vulnerability. A direct object reference means that an internal object such as a file or database key is exposed to the user. The problem with this is that the attacker can provide this reference and, if authorization is either not enforced (or is broken), the attacker can access or do things that they should be precluded from.

For example, the code has a download.php module that reads and lets the user download files, using a CGI parameter to specify the file name (e.g.download.php?file=something.txt). Either by mistake or due to laziness, the developer omitted authorization from the code. The attacker can now use this to download any system files that the user running PHP has access to, like the application code itself or other data left lying around on the server, like backups. Uh-oh.

Another common vulnerability example is a password reset function that relies on user input to determine whose password we’re resetting. After clicking the valid URL, an attacker can just modify the usernamefield in the URL to say something like “admin”.

Incidentally, both of these examples are things I myself have seen appearing often “in the wild”.

Prevention: Perform user authorization properly and consistently, and whitelist the choices. More often than not though, the whole problem can be avoided by storing data internally and not relying on it being passed from the client via CGI parameters. Session variables in most frameworks are well suited for this purpose.

In my experience, web servers and applications that have been misconfigured are way more common than those that have been configured properly. Perhaps this because there is no shortage of ways to screw up. Some examples:

1. Running the application with debug enabled in production.
2. Having directory listing enabled on the server, which leaks valuable information.
3. Running outdated software (think WordPress plugins, old PhpMyAdmin).
4. Having unnecessary services running on the machine.
5. Not changing default keys and passwords. (Happens way more frequently than you’d believe!)
6. Revealing error handling information to the attackers, such as stack traces.

Prevention: Have a good (preferably automated) “build and deploy” process, which can run tests on deploy. The poor man’s security misconfiguration solution is post-commit hooks, to prevent the code from going out with default passwords and/or development stuff built in.

This web security vulnerability is about crypto and resource protection. Sensitive data should be encrypted at all times, including in transit and at rest. No exceptions. Credit card information and user passwords should never travel or be stored unencrypted, and passwords should always be hashed. Obviously the crypto/hashing algorithm must not be a weak one – when in doubt, use AES (256 bits and up) and RSA (2048 bits and up).

And while it goes without saying that session IDs and sensitive data should not be traveling in the URLs and sensitive cookies should have the secure flag on, this is very important and cannot be over-emphasized.

Prevention:

• In transit: Use HTTPS with a proper certificate and PFS (Perfect Forward Secrecy). Do not accept anything over non-HTTPS connections. Have the secure flag on cookies.
• In storage: This is harder. First and foremost, you need to lower your exposure. If you don’t need sensitive data, shred it. Data you don’t have can’t be stolen. Do not store credit card information ever, as you probably don’t want to have to deal with being PCI compliant. Sign up with a payment processor such asStripe or Braintree. Second, if you have sensitive data that you actually do need, store it encrypted and make sure all passwords are hashed. For hashing, use of bcrypt is recommended. If you don’t use bcrypt, educate yourself on salting and rainbow tables.

And at the risk of stating the obvious, do not store the encryption keys next to the protected data. That’s like storing your bike with a lock that has the key in it. Protect your backups with encryption and keep your keys very private. And of course, don’t lose the keys!

This is simply an authorization failure. It means that when a function is called on the server, proper authorization was not performed. A lot of times, developers rely on the fact that the server side generated the UI and they think that the functionality that is not supplied by the server cannot be accessed by the client. It is not as simple as that, as an attacker can always forge requests to the “hidden” functionality and will not be deterred by the fact that the UI doesn’t make this functionality easily accessible. Imagine there’s an /adminpanel, and the button is only present in the UI if the user is actually an admin. Nothing keeps an attacker from discovering this functionality and misusing it if authorization is missing.

Prevention: On the server side, authorization must always be done. Yes, always. No exceptions or vulnerabilities will result in serious problems.

This is a nice example of a confused deputy attack whereby the browser is fooled by some other party into misusing its authority. A 3rd party site, for example, can make the user’s browser misuse it’s authority to do something for the attacker.

In the case of CSRF, a 3rd party site issues requests to the target site (e.g., your bank) using your browser with your cookies / session. If you are logged in on one tab on your bank’s homepage, for example, and they are vulnerable to this attack, another tab can make your browser misuse its credentials on the attacker’s behalf, resulting in the confused deputy problem. The deputy is the browser that misuses its authority (session cookies) to do something the attacker instructs it to do.

Consider this example:

Attacker Alice wants to lighten target Todd’s wallet by transfering some of his money to her. Todd’s bank is vulnerable to CSRF. To send money, Todd has to access the following URL:

http://example.com/app/transferFunds?amount=1500&destinationAccount=4673243243

After this URL is opened, a success page is presented to Todd, and the transfer is done. Alice also knows, that Todd frequently visits a site under her control at blog.aliceisawesome.com, where she places the following snippet:

img src="http://example.com/app/transferFunds?amount=1500&destinationAccount=4673243243" width="0" height="0" />

Upon visiting Alice’s website, Todd’s browser thinks that Alice links to an image, and automatically issues an HTTP GET request to fetch the “picture”, but this actually instructs Todd’s bank to transfer $1500 to Alice.

Incidentally, in addition to demonstrating the CSRF vulnerability, this example also demonstrates altering the server state with an idempotent HTTP GET request which is itself a serious vulnerability. HTTP GET requestsmust be idempotent (safe), meaning that they cannot alter the resource which is accessed. Never, ever, ever use idempotent methods to change the server state.

Fun fact: CSRF is also the method people used for cookie-stuffing in the past until affiliates got wiser.

Prevention: Store a secret token in a hidden form field which is inaccessible from the 3rd party site. You of course always have to verify this hidden field. Some sites ask for your password as well when modifying sensitive settings (like your password reminder email, for example), although I’d suspect this is there to prevent the misuse of your abandoned sessions (in an internet cafe for example).

The title says it all. I’d again classify this as more of a maintenance/deployment issue. Before incorporating new code, do some research, possibly some auditing. Using code that you got from a random person onGitHub or some forum might be very convenient, but is not without risk of serious web security vulnerability.

I have seen many instances, for example, where sites got owned (i.e., where an outsider gains administrative access to a system), not because the programmers were stupid, but because a 3rd party software remained unpatched for years in production. This is happening all the time with WordPress plugins for example. If you think they will not find your hidden phpmyadmininstallation, let me introduce you to dirbuster.

The lesson here is that software development does not end when the application is deployed. There has to be documentation, tests, and plans on how to maintain and keep it updated, especially if it contains 3rd party or open source components.

Prevention:

• Exercise caution. Beyond obviously using caution when using such components, do not be a copy-paste coder. Carefully inspect the piece of code you are about to put into your software, as it might be broken beyond repair (or in some cases, intentionally malicious).
• Stay up-to-date. Make sure you are using the latest versions of everything that you trust, and have a plan to update them regularly. At least subscribe to a newsletter of new security vulnerabilities regarding the product.

This is once again an input filtering issue. Suppose that the target site has a redirect.php module that takes a URL as a GETparameter. Manipulating the parameter can create a URL on targetsite.comthat redirects the browser to malwareinstall.com. When the user sees the link, they will see targetsite.com/blahblahblahwhich the user thinks is trusted and is safe to click. Little do they know that this will actually transfer them onto a malware drop (or any other malicious) page. Alternatively, the attacker might redirect the browser to targetsite.com/deleteprofile?confirm=1.

It is worth mentioning, that stuffing unsanitized user-defined input into an HTTP header might lead to header injection which is pretty bad.

Prevention: Options include:

• Don’t do redirects at all (they are seldom necessary).
• Have a static list of valid locations to redirect to.
• Whitelist the user-defined parameter, but this can be tricky.

I hope that I have managed to tickle your brain a little bit with this post and to introduce a healthy dose of paranoia and web security vulnerability awareness.

The core takeaway here is that age-old software practices exist for a reason and what applied back in the day for buffer overflows, still apply for pickled strings in Python today. Security helps you write correct(er) programs, which all programmers should aspire to.

Please use this knowledge responsibly, and don’t test pages without permission!

For more information and more specific attacks, have a look at:https://www.owasp.org/index.php/Category:Attack.

This post originally appeared in the Toptal blog: https://www.toptal.com/security/10-most-common-web-security-vulnerabilities

 To see how Morpheus can help you get more out of your MongoDB sign up for a demo today!

If a little bit of data is good, then a lot of data must be great, right? That's like saying if a cool breeze feels nice on a warm summer day, then a tornado will make you feel ecstatic.

Perhaps a better analogy for big data is a high-spirited champion racehorse: With the proper training and a talented jockey up, the thoroughbred can set course records, but minus the training and rider, the powerful animal would never make it into the starting gate.

To ensure your organization's big data plans stay on track, you need to dispel these 10 common misconceptions about the technology.

1. Big data simply means 'lots of data': At its core, big data describes how structured or unstructured data combine with social media analytics, IoT data, and other external sources to tell a "bigger story." That story may be a macro description of an organization's operation or a big-picture view that can't be captured using traditional analytic methods. A simple measure of the volume of data involved is insignificant from an intelligence-gathering perspective.

2. Big data needs to be clean as a whistle: In the world of business analytics, there is no such thing as "too fast." Conversely, in the IT world, there is no such thing as "garbage in, gold out." Just how clean is your data? One way to find out is to run your analytics app, which can identify weaknesses in your data collections. Once those weaknesses are addressed, run the analytics again to highlight the "cleaned up" areas.

3. All the human analysts will be replaced by machine algorithms: The recommendations of data scientists are not always implemented by the business managers on the front lines. Industry executive Arijit Sengupta states in a TechRepublic article that the proposals are often more difficult to put in place than the scientists project. However, relying too much on machine-learning algorithms can be just as challenging. Sengupta says machine algorithms tell you what to do, but they don't explain why you're doing it. That makes it difficult to integrate analytics with the rest of the company's strategic planning.

Predictive algorithms range from relatively simple linear algorithms to more sophisticated tree-based algorithms, and finally to extremely complex neural networks. Source: Dataiku, via Dataconomy

4. Data lakes are a thing: According to Toyota Research Institute data scientist Jim Adler, the huge storage repositories that some IT managers envision housing massive amounts of structured and unstructured data simply don't exist. Organizations don't indiscriminately dump all their data into one shared pool. The data is "carefully curated" in a department silo that encourages "focused expertise," Adler states. This is the only way to deliver the transparency and accountability required for compliance and other governance needs.

5. Algorithms are infallible prognosticators: Not long ago, there was a great deal of hype about the Google Flu Trends project, which claimed to predict the location of influenza outbreaks faster and more accurately than the U.S. Centers for Disease Control and other health information services. As the New Yorker's Michele Nijhuis writes in a June 3, 2017, article, it was thought that people's searches for flu-related terms would accurately predict the regions with impending outbreaks. In fact, simply charting local temperatures turned out to be a more accurate forecasting method.

Google's flu-prediction algorithm fell into a common big data trap: it made meaningless correlations, such as connecting high school basketball games and flu outbreaks because both occur during the winter. When data mining is operating on a massive set of data, it is more likely to encounter relationships among information that is statistically significant, yet entirely pointless. An example is linking the divorce rate in Maine with the U.S. per capita consumption of margarine: there is indeed a "statistically significant" relationship between the two numbers, despite the lack of any real-world significance.

6. You can't run big data apps on virtualized infrastructure: When "big data" first appeared on people's radar screens about 10 years ago, it was synonymous with Apache Hadoop. As VMware's Justin Murray writes in a May 12, 2017, article on Inside Big Data, the term now encompasses a range of technologies, from NoSQL (MongoDB, Apache Cassandra) to Apache Spark.

Critics previously questioned the performance of Hadoop on virtual machines, but Murray points out that Hadoop scales on VMs with performance comparable to bare metal, and it utilizes cluster resources more efficiently. Murray also blows up the misconception that the basic features of VMs require a storage area network (SAN). In fact, vendors frequently recommend direct attached storage, which offers better performance and lower costs.

7. Machine learning is synonymous with artificial intelligence: The gap between an algorithm that recognizes patterns in massive amounts of data and one that is able to formulate a logical conclusion based on the data patterns is more like a chasm. ITProPortal's Vineet Jain writes in a May 26, 2017, article that machine learning uses statistical interpretation to generate predictive models. This is the technology behind the algorithms that predict what a person is likely to buy based on past purchases, or what music they may like based on their listening history.

As clever as these algorithms may be, they are a far cry from achieving the goal of artificial intelligence, which is to duplicate human decision-making processes. Statistics-based predictions lack the reasoning, judgment, and imagination of humans. In this sense, machine learning may be considered a necessary precursor of true AI. Even the most sophisticated AI systems to date, such as IBM's Watson, can't provide the insights into big data that human data scientists deliver.

8. Most big data projects meet at least half their goals: IT managers know that no data-analysis project is 100-percent successful. When the projects involve big data, the success rates plummet, as shown by the results of a recent survey by NewVantage Partners (pdf). While 95 percent of the business leaders surveyed said their companies had engaged in a big data project over the past five years, only 48.4 percent of the projects had achieved "measurable results."

In fact, big data projects rarely get past the pilot stage, according to the results of Gartner research released in October 2016. The Gartner survey found that only 15 percent of big data implementations are ever deployed to production, which is relatively unchanged from the 14 percent success rate reported in the previous year's survey.

NewVantage Partners' Big Data Executive Survey indicates that fewer than half of all big data projects realize their goals, and "cultural" changes are the most difficult to achieve. Source: Data Informed

9. The rise of big data will reduce demand for data engineers: If a goal of your organization's big data initiatives is to minimize the need for data scientists, you may be in for an unpleasant surprise. The 2017 Robert Half Technology Salary Guide indicates that annual salaries for data engineers have jumped to an average between $130,000 and $196,000, while salaries for data scientists are now between $116,000 and $163,500 on average, and salaries for business intelligence analysts currently average from $118,000 to $138,750.

10. Employees and line managers will embrace big data with open arms: The NewVantage Partners survey found that 85.5 percent of the companies participating are committed to creating a "data-driven culture." However, the overall success rate of new data initiatives is only 37.1 percent. The three obstacles cited most often by these companies are insufficient organizational alignment (42.6 percent), lack of middle management adoption and understanding (41 percent), and business resistance or lack of understanding (41 percent).

The future may belong to big data, but realizing the technology's benefits will require a great deal of good old-fashioned hard work -- of the human variety.

When deciding to implement DevOps, you may not be sure when the best time to get started will be. Will other teams and individuals within your organization get on board immediately, or will it take some time and convincing? Can you get support from the top of the chain, and can they assist you in pushing forward a new way of operating so that people feel comfortable making the change? Is it imperative to make the change due to technical needs such as cloud offerings? All of these things can affect when the best time to implement DevOps is, so you will want to carefully decide what the best approach will be.

The main difference between doing DevOps right and otherwise is how the people interact with each other.  Shared knowledge, common goals, and desire to succeed are all traits of organizations wanting to do it right.

Source: Necco Ceresani for Xebia Labs

If everyone is already on board, then obviously it is a good time to go ahead as long as you have the necessary infrastructure and plan in place to do so. This, of course, is not usually the case unless you are in a particularly small organization where there is not a great deal of separation of teams from one another and you convey your message to as many different affected parties as possible much more quickly.

If you don’t have enough support yet, be sure to get a good plan with plenty of input from others to assist you in getting more people on board with the idea. This can also be helped by looking at the additional things below that can help you gain further support from others.

A vastly important and necessary factor is being able to get support from the top. If your leadership is not on board, it is unlikely you will be able to go forward with the implementation until they are. Again, if you have a good plan with input from others within the organization to help support your proposal, you will likely be able to get leadership on board much more easily.

Also, if leadership does not want to take an active role in getting people on board, it can certainly slow things down, as there may be some who feel no urgency to make any changes without a little support from the top. So, do your best to get the full support from leadership as this can get things moving forward much more quickly.

The most successful implementations start from practices you already know and use, scaled to organizational level and consistently improved with best practices. Big bangs – where everything is restarted from scratch – are bound to cause disruption and distraction.

Source: Mike Dunham for Scio

Whether or not you have support from the top, making others feel comfortable with the idea of changing things can go a long way toward getting a large number of people and team leaders on board quickly.

For example, if you simply show each team how DevOps would benefit them and save them time before implementation, then you stand a better chance of gaining their support. If you simply demand that teams start implementing new procedures on a specific date, you are likely to get much more negative results instead.

If you have leadership on board already, this can be enhanced if they will take part in this process and assist in showing how such a change will benefit each individual, each team, and the organization overall.

Of course, if there is a major need to change as quickly as possible, such as security issues with the current processes, then you may be able to convince others to get on board by explaining the issues that are causing this change to be necessary and timely. In such a case, it may be necessary to make the change anyway, so you will definitely want support from the top if this approach must be taken.

When doing so, as mentioned, you should already have a plan in place for the transition and get people as much information, training, and assistance as possible so that they are able to take an active part in the process. Many times, if people understand that they are helping to solve a potentially serious problem, they will be more willing to lend a supporting hand.

In the end, just be sure you have as many of these bases covered as you possibly can before deciding to make the switch. The more comfortable everyone is with making the change, the better a time it is to begin implementing DevOps in your organization!

If any IT job can be considered secure, you would think it would be that of a database administrator. After all, the U.S. Bureau of Labor Statistics Occupational Outlook Handbook forecasts an 11 percent increase in DBA employment from 2014 to 2024. That's a faster rate than the average for all occupations, and just a tick below the 12 percent growth rate projected by the agency for all computer jobs.

Before any of you DBAs out there get too comfortable about your employment prospects, keep in mind that the database skillset continues the transformation triggered by the rise of cloud computing and database as a service (DBaaS). According to the BLS numbers, DBA employment at cloud service providers will grow by 26 percent in the decade ending in 2024.

The shifting emphasis to cloud databases hasn't had a great impact on the skills companies look for in their new DBAs. While database administration is one of the top ten "hot skills" identified in Computerworld's Forecast 2017 survey of in-demand tech skills, SQL programmers continue to be the most sought-after group. The survey found that 25 percent of companies plan to hire DBAs this year.

IT managers place database administration as one of the top 10 hot skills in 2017: 25 percent of the companies surveyed plan to hire a DBA in 2017. Source: Computerworld

One change noted in the 2017 survey is an increased focus on DBAs who "understand the user experience," according to Michelle Beveridge, CIO for adventure travel firm Intrepid Group. DBAs need to look beyond data rules, mandatory input requirements, and data structures to consider first and foremost the business processes behind the data collection. Finding DBAs with these skills will continue to be a challenge, Beveridge states.

Slow pace of change in languages, tools benefits experienced DBAs, programmers

A look at the most recent DB-Engines ranking of DBMSs by popularity indicates the staying power of traditional relational databases: Oracle, MySQL, and Microsoft SQL Server continue to dominate the rankings, as they have for years. As of the June 2017 numbers, the PostgreSQL relational DBMS took over the fourth spot (year-to-year) from the MongoDB document store, while Cassandra was rated eighth (down one from the year-earlier ranking), and Redis was ninth (up one place from the June 2016 scores).

The continuing popularity of old favorites is also evident in the June 2017 RedMonk Programming Language Rankings, which are based on rankings from GitHub and Stack Overflow. JavaScript and Java have held down the top two spots since the inception of the rankings in 2012. Python, PHP, and C# have traded the third, fourth, and fifth positions on the list for almost as long. Another long-time favorite, C++, is holding steady in the sixth position, while Ruby dropped to eighth after peaking in the fourth spot in the third quarter of 2013.

The most upwardly mobile language on the list is Kotlin, which is ranked only 46th, but the language has moved up from number 65 in the December 2016 rankings. Most of the bump is attributed to Google's decision in May 2017 to make Kotlin the company's alternative to Swift, which is currently rated 11th. The language is expected to continue its climb up the rankings as more Android developers experiment with Kotlin as they create new apps.

Enterprise repositories find their way to the public cloud

Data warehouses represent one of the last bastions of in-house data centers. A new class of public cloud data repositories is challenging the belief that warehouses need to reside on the premises. TechTarget's Trevor Jones writes in a June 8, 2017, article that services such as Amazon Redshift, Google Cloud Platform BigQuery, and Microsoft Azure SQL Data Warehouse provide greater abstraction and integration with related services. This makes it simpler for managers to explore the organization's deep pools of data alongside traditional data warehouses.

Choosing a database depends in large part on the size of the data you need to accommodate - the more data you have, the more likely you'll need a non-relational database. Source: Stephen Levin, via Segment

The goal of such services is to enhance business intelligence by tapping a range of cloud services hosting structured and unstructured data. However, the challenges in realizing this goal are formidable, particularly for enterprises. Much of a company's existing structured data must be cleaned or rewritten for the transition to cloud platforms, and it isn't unusual for enterprises to have workloads in several different cloud services.

One company in the process of transitioning to a cloud data warehouse is the New York Times, which previously built its own Hadoop cluster and used data warehouses from Informatica, Oracle, AWS, and other vendors. This setup left much of the company's data "too siloed and too technical," according to Jones. The Times is now transitioning to Google Cloud Platform as the sole receptacle for all of its warehoused data, primarily as a way to put powerful analytical tools in the hands of users.

Laying the groundwork for real-time analytics

A technology likely to have a great impact on DBAs in coming years is real-time analytics, which is also called streaming analytics. Dataversity defines stream processing as analyzing and acting on data in real time by applying continuous queries. Applications connect to external data sources, integrate analytics in the app "flow," and update external databases with the processed information.

While descriptive, predictive, and prescriptive analytics perform batch analysis on historical data, streaming analytics evaluate and visualize data in real time. This facilitates operational decision-making related to business processes, transactions, and production. It also allows current and historical data to be reported at the same time, which makes it possible to display via a dashboard changes in transactional data sets in real time.

The components of a real-time streaming analytics architecture include real-time and historical data combined in an event engine from which real-time actions are taken and displayed on a dashboard. Source: Paul Stanton, via MarTech Advisor

Many obstacles must be overcome to realize the benefits of real-time analytics. For example, a great number of organizations rely on Hadoop for analyzing their large stores of historical data, but Hadoop can't accommodate streaming, real-time data. Alternatives include MongoDB, Apache Flink, Apache Samza, Spark Streaming, and Storm. Also, real-time data flows are likely to overwhelm existing business processes, and the costs of faulty analysis increase exponentially.

The more insights you can gain from your organization's data resources, and the faster those insights can be applied to business decisions, the more value you can squeeze out of your information systems. Putting practical business intelligence in the hands of managers when and where they need it is the reward that lets DBAs know they're contributing directly to their company's success.

Note: When we recently launched, we were thrilled to have SQL Guru Pinal Dave give Morpheus a spin. It turns out that he had a great experience, and as he is indeed an SQLAuthority, we thought we'd share his post here as well. Without further delay, Pinal shares his thoughts below:

I love my weekend projects. Everybody does different activities in their weekend – like traveling, reading or just nothing. Every weekend I try to do something creative and different in the database world. The goal is I learn something new and if I enjoy my learning experience I share with the world. This weekend, I decided to explore Cloud Database As A Service – Morpheus. In my career I have managed many databases in the cloud and I have good experience in managing them.

I should highlight that today’s applications use multiple databases from SQL for transactions and analytics, NoSQL for documents, In-Memory for caching to Indexing for search. Provisioning and deploying these databases often require extensive expertise and time. Often these databases are also not deployed on the same infrastructure and can create unnecessary latency between the application layer and the databases. Not to mention the different quality of service based on the infrastructure and the service provider where they are deployed.

Moreover, there are additional problems that I have experienced with traditional database setup when hosted in the cloud:

• Database provisioning & orchestration
• Slow speed due to hardware issues
• Poor Monitoring Tools
• High Network Latency

Now if you have a great software and expert network engineer, you can continuously work on above problems and overcome them. However, not every organization have the luxury to have top notch experts in the field. Now above issues are related to infrastructure, but there are a few more problems which are related to software/application as well.

Here are the top three things which can be problems if you do not have application expert: 

• Replication and Clustering
• Simple provisioning of the hard drive space
• Automatic Sharding

Well, Morpheus looks like a product build by experts who have faced similar situation in the past. The product pretty much addresses all the pain points of developers and database administrators.

What is different about Morpheus is that it offers a variety of databases from MySQL, MongoDB, ElasticSearch to Redis as a service. Thus users can pick and chose any combination of these databases. All of them can be provisioned in a matter of minutes with a simple and intuitive point and click user interface. The Morpheus cloud is built on Solid State Drives (SSD) and is designed for high-speed database transactions. Inaddition it offers a direct link to Amazon Web Services to minimize latency between the application layer and the databases.

Here are the few steps on how one can get started with Morpheus. Follow along with me. First go to http://www.gomorpheus.com and register for a new and free account.

Step 1: Signup

It is very simple to signup for Morpheus.

Step 2: Select your database

I use MySQL for my daily routine, so I have selected MySQL. Upon clicking on the big red button to add Instance, it prompted a dialogue of creating a new instance.

Step 3: Create User

Now we just have to create a user in our portal which we will use to connect to a database hosted at Morpheus. Click on your database instance and it will bring you to User Screen. Over here you will notice once again a big red button to create a new user. I created a user with my first name.

Step 4: Configure Your MySQL Client

I used MySQL workbench and connected to MySQL instance, which I had created with an IP address and user.

That’s it! You are connecting to MySQL instance. Now you can create your objects just like you would create on your local box. You will have all the features of the Morpheus when you are working with your database.

Dashboard

While working with Morpheus, I was most impressed with its dashboard. In future blog posts, I will write more about this feature. Also with Morpheus you use the same process for provisioning and connecting with other databases: MongoDB, ElasticSearch and Reddis.

[TL:DR] The theft of hundreds of millions of user IDs, passwords, and email addresses was made possible by a database programming technique called dynamic SQL, which makes it easy for hackers to use SQL injection to gain unfettered access to database records. To make matters worse, the dynamic SQL vulnerability can be avoided by using one of several simple programming alternatives.

How is it possible for a simple hacking method which has been publicized for as many as 10 years to be used by Russian cybercriminals to amass a database of more than a billion stolen user IDs and passwords? Actually, the total take by the hackers in the SQL injection attacks revealed earlier this month by Hold Security was 1.2 billion IDs and passwords, along with 500 million email addresses, according to an article written by Nicole Perlroth and David Gelles in the August 5, 2014, New York Times.

Massive data breaches suffered by organizations of all sizes in recent years can be traced to a single easily preventable source, according to security experts. In an interview with IT World Canada's Howard Solomon, security researcher Johannes Ullrich of the SANS Institute blames an outdated SQL programming technique that continues to be used by some database developers. The shocker is that blocking such malware attacks is as easy as using two or three lines of code in place of one. Yes, according to Ullrich, it's that simple.

The source of the vulnerability is dynamic SQL, which allows developers to create dynamic database queries that include user-supplied data. The Open Web Application Security Project (OWASP) identifies SQL, OS, LDAP, and other injection flaws as the number one application security risk facing developers. An injection involves untrusted data being sent to an interpreter as part of a command or query. The attacker's data fools the interpreter into executing commands or accessing data without authentication.

According to OWASP, injections are easy for hackers to implement, difficult to discover via testing (but not by examining code), and potentially severely damaging to businesses.

The OWASP SQL Injection Prevention Cheat Sheet provides a primer on SQL injection and includes examples of unsafe and safe string queries in Java, C# .NET, and other languages.

An example of an unsafe Java string query (top) and a safe Java PreparedStatement (bottom).

Dynamic SQL lets comments be embedded in a SQL statement by setting them off with hyphens. It also lets multiple SQL statements to be strung together, executed in a batch, and used to query metadata from a standard set of system tables, according to Solomon.

Three simple programming approaches to SQL-injection prevention

OWASP describes three techniques that prevent SQL injection attacks. The first is use of prepared statements, which are also referred to as parameterized queries. Developers must first define all the SQL code, and then pass each parameter to the query separately, according to the OWASP's prevention cheat sheet. The database is thus able to distinguish code from data regardless of the user input supplied. A would-be attacker is blocked from changing the original intent of the query by inserting their own SQL commands.

The second prevention method is to use stored procedures. As with prepared statements, developers first define the SQL code and then pass in the parameters separately. Unlike prepared statements, stored procedures are defined and stored in the database itself, and subsequently called from the application. The only caveat to this prevention approach is that the procedures must not contain dynamic SQL, or if it can't be avoided, then input validation or another technique must be employed to ensure no SQL code can be injected into a dynamically created query.

The last of the three SQL-injection defenses described by OWASP is to escape all user-supplied input. This method is appropriate only when neither prepared statements nor stored procedures can be used, whether because doing so would break the application or render its performance unacceptable. Also, escaping all user-supplied input doesn't guarantee your application won't be vulnerable to a SQL injection attack. That's why OWASP recommends it only as a cost-effective way to retrofit legacy code. 

All databases support one or more character escaping schemes for various types of queries. You could use an appropriate escaping scheme to escape all user-supplied input. This prevents the database from mistaking the user-supplied input for the developer's SQL code, which in turn blocks any SQL injection attempt.

The belt-and-suspenders approach to SQL-injection prevention

Rather than relying on only one layer of defense against a SQL injection attack, OWASP recommends a layered approach via reduced privileges and white list input validation. By minimizing the privileges assigned to each database account in the environment, DBAs can reduce the potential damage incurred by a successful SQL injection breach. Read-only accounts should be granted access only to those portions of database tables they require by creating a specific view for that specific level of access. Database accounts rarely need create or delete access, for example. Likewise, you can restrict the stored procedures certain accounts can execute. Most importantly, according to OWASP, minimize the privileges of the operating system account the database runs under. MySQL and other popular database systems are set with system or root privileges by default, which likely grants more privileges than the account requires.

Adopting the database-as-a-service model limits vulnerability

Organizations of all sizes are moving their databases to the cloud and relying on services such as Morpheus to ensure safe, efficient, scalable, and affordable management of their data assets. Morpheus supports MongoDB, MySQL, Redis, ElasticSearch, and other DB engines. The service's real-time monitoring lets you analyze and optimize the performance of database applications.

In addition to 24/7 monitoring of your databases, Morpheus provides automatic backup, restoration, and archiving of your data, which you can access securely via a VPN connection. The databases are stored on Morpheus's solid-state drives for peak performance and reliability.  

TL; DR: The tremendous growth predicted for the open-source Hadoop architecture for data analysis is driven by the mind-boggling increase in the amount of structured and unstructured data in organizations, and the need for sophisticated, accessible tools to extract business and market intelligence from the data. New cloud services such as Morpheus let organizations of all sizes realize the potential of Big Data analysis.

The outlook is rosy for Hadoop -- the open-source framework designed to facilitate distributed processing of huge data sets. Hadoop is increasingly attractive to organizations because it delivers the benefits of Big Data while avoiding infrastructure expenses.

A recent report from Allied Market Research concludes that the Hadoop market will realize a compound annual growth rate of 58.2 percent from 2013 to 2020, to a total value of $50.2 billion in 2020, compared to $1.5 billion in 2012.

Allied Market Research forecasts a $50.2 billion global market for Hadoop services by the year 2020.

Just how "big" is Big Data? According to IBM, 2.5 quintillion bytes of data are created every day, and 90 percent of all the data in the world was created in the last two years. Realizing the value of this huge information store requires data-analysis tools that are sophisticated enough, cheap enough, and easy enough for companies of all sizes to use.

Many organizations continue to consider their proprietary data too important a resource to store and process off premises. However, cloud services now offer security and availability equivalent to that available for in-house systems. By accessing their databases in the cloud, companies also realize the benefits of affordable and scalable cloud architectures.

The Morpheus database-as-a-service offers the security, high availability, and scalability organizations require for their data-intelligence operations. Performance is maximized through Morpheus's use of 100-percent bare-metal SSD hosting. The service offers ultra-low latency to Amazon Web Services and other peering points and cloud hosting platforms.

The Nuts and Bolts of Hadoop for Big Data Analysis

The Hadoop architecture distributes both data storage and processing to all nodes on the network. By placing the small program that processes the data in the node with the much larger data sets, there's no need to stream the data to the processing module. The processor splits its logic between a map and a reduce phase. The Hadoop scheduling and resource management framework executes the map and reduce phases in a cluster environment.

The Hadoop Distributed File System (HDFS) data storage layer uses replicas to overcome node failures and is optimized for sequential reads to support large-scale parallel processing. The market for Hadoop really took off when the framework was extended to support the Amazon Web Services S3 and other cloud-storage file systems.

Adoption of Hadoop in small and midsize organizations has been slow despite the framework's cost and scalability advantages because of the complexity of setting up and running Hadoop clusters. New services do away with much of the complexity by offering Hadoop clusters that are managed and ready to use: there's no need to configure or install any services on the cluster nodes.

Netflix data warehouse combines Hadoop and Amazon S3 for infinite scalability

For its petabyte-scale data warehouse, Netflix chose Amazon's Storage Service (S3) over the Hadoop Distributed File System for the cloud-based service's dynamic scalability and limitless data and computational power. Netflix collects data from billions of streaming events from televisions, computers, and mobile devices.

With S3 as its data warehouse, Hadoop clusters with hundreds of nodes can be configured for various workloads, all able to access the same data. Netflix uses Amazon's Elastic MapReduce distribution of Hadoop and has developed its own Hadoop Platform as a Service, which it calls Genie. Genie lets users submit jobs from Hadoop, Pig, Hive, and other tools without having to provision new clusters or install new clients via RESTful APIs.

The Netflix Hadoop-S3 data warehouse offers unmatched elasticity in terms of data and computing power in a widely distributed network.

There is clearly potential in combining Hadoop and cloud services, as Wired's Marco Visibelli explains in an August 13, 2014, article. Visibelli describes how companies leverage Big Data for forecasting by scaling from small projects via Amazon Web Services and scaling up as their small projects succeed. For example, a European car manufacturer used Hadoop to combine several supplier databases into a single 15TB database, which saved the company $16 million in two years.

Hadoop opens the door to Big Data for organizations of all sizes. Projects that leverage the scalability, security, accessibility, and affordability of cloud services such as Morpheus's database as a service have a much greater chance of success.

TL;DR: As IT managers gain confidence in the reliability and security of cloud services, it becomes more difficult for them to ignore the cloud's many benefits for all their business's operations. Companies have less hardware to purchase and maintain, they spend only for the storage and processing they need, and they can easily monitor and manage their applications. With the Morpheus database as a service you get all of the above running on a high-availability network that features 24-7 support.

Give any IT manager three wishes and they'll probably wish for three fewer things to worry about. How about 1) having less hardware to buy and manage, 2) having to pay for only the storage and processing you need, and 3) being able to monitor and test applications from a single easy-to-use console?

Knowing the built-in cynicism of many data-center pros, they're likely to scoff at your offer, or at least suspect that it can't be as good as it sounds. That's pretty much the reception cloud services got in the early days, circa 2010.

An indication of IT's growing acceptance of cloud services for mainstream applications is KPMG's annual survey of 650 enterprise executives in 16 countries about their cloud strategies. In the 2011 survey, concerns about data security, privacy, and regulatory compliance were cited as the principal impediments to cloud adoption in large organizations.

According to the results of the most recent KPMG cloud survey, executives now consider cloud integration challenges and control of implementation costs as their two greatest concerns. There's still plenty of fretting among executives about the security of their data in the cloud, however. Intellectual property theft, data loss/privacy, and system availability/business continuity are considered serious problems, according to the survey.

Executives rate such cloud-security challenges as intellectual property theft, data loss, and system availability greater than 4 on a scale of 1 (not serious) to 5 (very serious). Credit: KPMG

Still, security concerns aren't dissuading companies from adopting cloud services. Executives told KPMG that in the next 18 months their organizations planned cloud adoption in such areas as sourcing and procurement; supply chain and logistics; finance, accounting and financial management; business intelligence and analytics; and tax.

Cloud 'migration' is really a 'transformation'

Three business trends are converging to make the cloud an integral part of the modern organization: the need to collect, integrate, and analyze data from all internal operations; the need to develop applications and business processes quickly and inexpensively; and the need to control and monitor the use of data resources that are no longer stored in central repositories.

In a September 2, 2014, article on Forbes.com, Robert LeBlanc explains that cloud services were initially perceived as a way to make operations more efficient and less expensive. But now organizations see the cloud architecture as a way to innovate in all areas of the company. Business managers are turning to cloud services to integrate big data, mobile computing, and social media into their core processes.

Mobile and collaboration are leading the transition in organizations away from on-site management and toward cloud platforms. Credit: Ventana Research

George Washington University discovered first-hand the unforeseen benefits of its shift to a cloud-based data strategy. Zaid Shoorbajee describes in the March 3, 2014, GW Hatchet student newspaper how a series of campus-wide outages motivated the university to migrate some operations to cloud services. The switch saved the school $700,000 and allowed its IT staff to focus more on development and less on troubleshooting.

The benefits the school realized from the switch extend far beyond IT, however. Students now have the same "consumer and social experience" they've become accustomed to in their private lives through Google, iTunes, and similar services, according to a university spokesperson.

Four approaches to cloud application integration

Much of the speed, efficiency, and agility of cloud services can be lost when organizations become bogged down in their efforts to adapt legacy applications and processes. In a TechTarget article (registration required), Amy Reichert presents four approaches to cloud application integration. The process is anything but simple, due primarily to the nature of the applications themselves and the need to move data seamlessly and accurately between applications to support business processes.

One of the four techniques is labeled integration platform as a service (iPaas), in which the cloud service itself provides integration templates featuring such tools as connectors, APIs, and messaging systems. Organizations then customize and modify the templates to meet their specific needs.

In cloud-to-cloud integration, the organization's cloud applications have an integration layer built in to support any required data transformations, as well as encryption and transportation. The cloud-to-integrator-to-cloud model relies on the organization's existing middleware infrastructure to receive, convert, and transport the data between applications.

Finally, the hybrid integration approach keeps individual cloud apps separate but adds an integration component to each. This allows organizations to retain control over the data, maximize its investment in legacy systems, and adopt cloud services at the company's own pace.

Regardless of your organization's strategy for adopting and integrating cloud applications, the Morpheus database as a service can play a key role by providing a flexible, secure, and reliable platform for monitoring and optimizing database applications. Morpheus's SSD-backed infrastructure ensures lightning fast performance, and direct patches into EC2 offer ultra-low latency.

Morpheus protects your data via secure VPC connections and automatic backups, replication, and archiving. The service supports ElasticSearch, MongoDB, MySQL, and Redis, as well as custom storage engines. Create your free database during the beta period.

TL;DR: At one time, organizations planning their cloud strategy adopted an either-or approach: Either store and manage data on a secure private cloud, or opt for the database-as-a-service model of the public cloud. Now companies are realizing the benefits of both options by adopting a multicloud strategy that places individual applications on the platform that best suits them.

In IT's never-ending quest to improve database performance and reduce costs, a new tactic has surfaced: multicloud. Rather that process all database queries on either the private cloud or public cloud, shift the processing to the platform best able to handle it in terms of speed and efficiency.

InfoWorld's David Linthicum explains in an August 5, 2014, article that a multicloud architecture "gives those who manage large distributed databases the power to use only the providers who offer the best and most cost-effective service -- or the providers who are best suited to their database-processing needs."

Managing the resulting complexity isn't as daunting as it may sound, according to Linthicum. In fact, a cloud-management system could soon become a requirement for IT departments of all sizes. Product lifecycle management (PLM) expert Oleg Shilovitsky claims in an August 5, 2014, article on BeyondPLM.com that three trends are converging to make distributed database architectures mandatory.

The first trend is the tsunami of data that is overwhelming information systems and pushing traditional database architectures to their physical limits. The second trend is the increasingly distributed nature of organizations, which are adopting a design-anywhere, build-anywhere philosophy. The third trend is the demand among users for ever-faster performance on many different platforms to keep pace with the changes in the marketplace.

Multicloud: More than simply pairing public and private

In a July 12, 2013, article, InfoWorld's Linthicum compared the process of adopting a multicloud strategy to the transition a decade or more ago to distributed internal systems customized to the specific demands of the business. A key to managing the increased complexity of multicloud systems is carefully choosing your service provider to ensure a good fit between their offerings and your company's needs.

Three key considerations in this regard are security, accessibility, and scalability. These are three areas where the Morpheus database-as-a-service shines. In addition to lightning-fast SSD-based infrastructure that increases IOPs by 100 times, Morpheus provides real-time monitoring for identifying and optimizing database queries that are impeding database performance.

Morpheus offers ultra-low latency to leading Internet peering points and cloud hosts. Additionally, fault tolerance, disaster recovery, and automated backups make Morpheus a unique Database as a service. You connect to your databases via secure VPC. Visit the Morpheus site for pricing information or to create a free account during the beta period.

Mixing old and new while maximizing adaptability

Businesses of all types and sizes are emphasizing the ability to shift gears quickly in anticipation of industry trends. No longer can you simply react to market changes: You must be there ahead of the competition.

A principal benefit of the multicloud database architecture is flexibility. In an August 25, 2014, article on Forbes.com, IBM's Jeff Borek highlights the ability of multicloud databases to leverage existing IT infrastructure while realizing the agility, speed, and cost savings of cloud services.

A typical multicloud approach is use of the private cloud as a point-of-control interface to public cloud services. MSPMentor's Michael Brown describes such an architecture in an August 27, 2014, article.

Many companies use a private cloud to ensure regulatory compliance for storing health, financial, and other sensitive data. In such systems, the private cloud may serve as the gateway to the public cloud in a two-tier structure. In addition to providing a single interface for users, the two levels allow applications and processes to be customized for best fit while keeping sensitive data secure.

A multicloud-application prototype: Managing multiple application servers

There's no denying that managing a distributed database system is more complicated than maintaining the standard top-down RDBMS of yesteryear. In a July 23, 2013, article on GitHub, German Ramos Garcia presents a prototype multicloud application development model based on the Hydra service. The model addresses much of the complexity entailed in managing multiple application servers.

The web application is first divided into static elements (images, Javascript, static HTML, etc.), dynamic elements on a backend server, and a database to support the backend servers.

A prototype multicloud application architecture separates static, dynamic, and database-support servers.

The distributed architecture must provide mechanisms for controlling the various servers, balancing traffic between servers, and recovering from failures. It must also control sessions between servers and determine where to store application data.

An alternative approach to multicloud management is presented by Mauricio J. Rojas in a blog post from March 25, 2014. The model Rojas proposes is a mash-up of management tools from many different cloud services.

Management tools for distributed cloud-based databases should focus on user needs and offer best of breed from various providers.

Rojas recommends creating a single set of management components for both the public and private clouds. This allows you to "create the same conditions in both worlds" and move seamlessly between the public and private domains.

In addition to security, important considerations in developing a multicloud management system are auto-scaling and high availability. With the Morpheus database-as-a-service, you're covered in all three areas right out of the box--even Pinal Dave, the SQL Authority uses Morpheus.  Make Morpheus a key element of your multicloud strategy.

TL;DR: Following several high-profile development disasters, government IT departments have received a mandate to change their default app-development approach from the traditional top-down model to the agile, iterative, test-centric methodology favored by leading tech companies. While previous efforts to dynamite the entrenched, moribund IT-contracting process have crashed in flames, analysts hold out hope for the new 18F and U.S. Digital Service initiatives. Given the public's complete lack of faith in the government's ability to provide digital services, failure is simply not an option.

Can Silicon Valley save the federal government from itself? That's the goal of former U.S. Chief Technology Officer Todd Park, who relocated to California this summer and set about recruiting top-tier application developers from the most innovative tech companies on the planet to work for the government.

As Wired's Steven Levy reports in an August 28, 2014, article, Park hopes to appeal to developers' sense of patriotism. "America needs you," Levy quotes Park telling a group of engineers at the Mozilla Foundation headquarters. A quick review of recent federal-government IT debacles demonstrates the urgency of Park's appeal.

Start with the $300 million spent over the past six years by the Social Security Administration on a disability-claim filing system that remains unfinished. Then check out the FBI's failed Virtual Case File case-management initiative that had burnt through $600 million before being replaced by the equally troubled Sentinel system, as Jason Bloomberg explains in an August 22, 2012, CIO article.

But the poster child of dysfunctional government app development is HealthCare.gov., which Park was brought in to save after its spectacularly failed launch in October 2013. For their $300 million investment, U.S. taxpayers got a site that took eight seconds to respond to a mouse click and crashed so often that not one of the millions of people visiting the site on its first day of operation was able to complete an application.

Healthcare.gov's performance in the weeks after its launch highlight what can happen when a $300 million development project proceeds with no one in the driver's seat. Credit: The Verge

The dynamite approach to revamping government IT processes

Just months before HealthCare.gov's epic crash-and-burn, Park had established the Presidential Innovation Fellows program to attract tech professionals to six-month assignments with the government. The program was envisioned as a way to seed government agencies with people who could introduce cutting-edge tools and processes to their development efforts. After initial successes with such agencies as Medicare and Veterans Affairs, the group turned its attention to rescuing HealthCare.gov -- and perhaps the entire Affordable Care Act.

The source of the site's problems quickly became obvious: the many independent contractors assigned to portions of the site worked in silos, and no single contractor was responsible to ensure the whole shebang actually worked. Even as the glitches stacked up following the failed launch, contractors continued to work on new "features" because they were contractually required to meet specific goals.

The culprit was the federal contracting process. Bureaucrats farmed out contracts to cronies and insiders, whose only motivation was to be in good position to win the next contract put up for bid, according to Levy. Park's team of fixers was met with resistance at every turn despite being given carte blanche to ignore every rule of government development and procurement.

With persistence and at least one threat of physical force, the ad-hoc team applied a patchwork of monitoring, testing, and debugging tools that got the site operational. By April 2014, HealthCare.gov had achieved its initial goal of signing up 8 million people for medical insurance.

How an agile-development approach could save democracy

The silver lining of the HealthCare.gov debacle is the formation of two new departments charged with bringing an agile approach to government app development. The General Services Administration's 18F was established earlier this year with a mandate to "fail fast" rather than follow the standard government-IT propensity to fail big.

As Tech President's Alex Howard describes in an August 14, 2014, article, 18F is assisting agencies as they develop free, open-source services offered to the public via GitHub and other open-source repositories. Perhaps an even-bigger shift in attitude by government officials is the founding last month of the U.S. Digital Service, which is modeled after a successful U.K. government app-development program.

To help agencies jettison their old development habits in favor of modern approaches, the White House released the Digital Services Playbook that provides 13 "plays" drawn from successful best practices in the private and public sectors. Two of the plays recommend deploying in a flexible hosting environment and automating testing and deployment.

The government's Digital Services Playbook calls for agencies to implement modern development techniques such as flexible hosting and automated testing.

That's precisely where the Morpheus database-as-a-service (DBaas) fits into the government's plans. Morpheus lets users spin up a new database instance in seconds -- there's no need to wait for lengthy IT approval to procure and provision a new DB. Instead it's all done in the cloud within seconds.

In addition, users' core elastic, scalable, and reliable DB infrastructure is taken care for them. Developers can focus on building the core functionality of the app rather than having to spend their time making the infrastructure reliable and scalable. Morpheus delivers continuous availability, fault tolerance, fail over, and disaster recovery for all databases running on its service. Last but definitely not least, it's cost efficient for users to go with Morpheus: there's no upfront setup cost, and they pay only for actual usage.

The Morpheus cloud database as a service (DBaaS) epitomizes the goals of the government's new agile-development philosophy. The service's real-time monitoring makes continuous testing a fundamental component of database development and management. Morpheus's on-demand scalability ensures that applications have plenty of room to grow without incurring large up-front costs. You get all this plus industry-leading performance, VPN security, and automatic backups, archiving, and replication.

Government IT gets the green light to use cloud app-development services

As groundbreaking as the Digital Services Playbook promises to be for government IT, another publication released at the same time may have an even-greater positive impact on federal agencies. The TechFAR Handbook specifies how government contractors can support an "iterative, customer-driven software development process."

Tech President's Howard quotes Code for America founder Jen Pahlka stating that the handbook makes it clear to government IT staff and contractors alike that "agile development is not only perfectly legal, but [is] in fact the default methodology."

Critics point out that this is not the government's first attempt to make its application development processes more open and transparent. What's different this time is the sense of urgency surrounding efforts such as 18F and the U.S. Digital Service. Pahlka points out that people have lost faith in the government's ability to provide even basic digital services. Pahlka is quoted in a July 21, 2014, Government Technology interview by Colin Wood and Jessica Mulholland as stating, "If government is to regain the trust and faith of the public, we have to make services that work for users the norm, not the exception."

TL;DR: Securing your company's cloud-based assets starts by applying tried-and-true data-security practices modified to address the unique characteristics of virtual-network environments. Cloud services are slowly gaining the trust of IT managers who are justifiably hesitant to extend the security perimeters to accommodate placing their company's critical business assets in the cloud.

The fast pace of technological change doesn't faze IT pros, who live the axiom "The more things change, the more they stay the same." The solid security principles that have protected data centers for generations apply to securing your organization's assets that reside in the cloud. The key is to anticipate the new threats posed by cloud technology -- and by cyber criminals who now operate with a much higher level of sophistication.

In a September 18, 2014, article, ZDNet's Ram Lakshminarayanan breaks down the cloud-security challenge into four categories: 1) defending against cloud-based attacks by well-funded criminal organizations 2) unauthorized access and data breaches that use employees' stolen or compromised mobile devices 3) maintenance and monitoring of cloud-based APIs, and 4) ensuring compliance with the growing number and complexity of government regulations.

IT departments are noted for their deliberate approach to new technologies, and cloud-based data services are no different. According to a survey published this month by the Ponemon Institute of more than 1,000 European data-security practitioners (pdf), 64 percent believe their organization's use of cloud services reduces their ability to protect sensitive information.

The survey, which was sponsored by Netskope, blames much of the distrust on the cloud multiplier effect: IT is challenged to track the increasing number and type of devices connecting to the company's networks, as well as the cloud-hosted software employees are using, and the business-critical applications being used in the "cloud workspace."

Building trust between cloud service providers and their IT customers

No IT department will trust the organization's sensitive data to a service that fails to comply with privacy and data-security regulations. The Ponemon survey indicates that cloud services haven't convinced their potential customers in Europe of their trustworthiness: 72 percent of respondents strongly disagreed, disagreed, or were uncertain whether their cloud-service providers were in full compliance with privacy and data-security laws.

Data-security executives remain leery of cloud services' ability to secure their organization's critical business data. Credit: Ponemon Institute

Even more troubling for cloud service providers is the survey finding that 85 percent of respondents strongly disagreed, disagreed, or weren't sure whether their cloud service would notify them immediately in the event of a data breach that affected their company's confidential information or intellectual property.

The Morpheus database-as-a-service puts data security front and center by offering VPN connections to your databases in addition to online monitoring and support. Your databases are automatically backed up, replicated, and archived on the service's SSD-backed infrastructure.

Morpheus also features market-leading performance, availability, and reliability via direct connections to EC2 and colocation with the fastest peering points available. The service's real-time monitoring lets you identify and optimize the queries that are slowing your database's performance. Visit the Morpheus site for pricing information and to sign up for a free account.

Overcoming concerns about cloud-service security

Watching your data "leave the nest" can be difficult for any IT manager. Yet cloud service providers offer a level of security at least on par with that of their on-premises networks. In a September 15, 2014, article on Automated Trader, Bryson Hopkins points out that Amazon Web Services and Microsoft Azure are two of the many public cloud services that comply with Service Organization Control (SOC), HIPPA, FedRAMP, ISO 27001, and other security standards.

The SANS Institute's Introduction to Securing a Cloud Environment (pdf) explains that despite the cloud's increased "attack surface" when compared with in-house servers, the risk of cloud-based data being breached is actually less than that of losing locally hosted data. Physical and premises security are handled by the cloud service but can be enhanced by applying a layered approach to security that uses virtual firewalls, security gateways, and other techniques.

Cloud services avoid resource contention and other potential problems resulting from multi-tenancy by reprovisioning virtual machines, overprovisioning to crowd out other tenants, and using fully reserved capacities.

Another technique for protecting sensitive data in multi-tenant environments is to isolate networks by configuring virtual switches or virtual LANs. The virtual machine and management traffic must be isolated from each other at the data link layer (layer 2) of the OSI model.

The key to protecting sensitive data in a multi-tenant cloud environment is to isolate virtual machine and management traffic at the data link layer. Credit: SANS Institute

In a June 27, 2014, article on CloudPro, Davey Winder brings the issue of cloud security full circle by highlighting the fact that the core principles are the same as for other forms of data security: an iron-clad policy teamed with encryption. The policy must limit privileged-user access by the service's employees and provide a way for customers to audit the cloud network.

One way to compare in-house data management and cloud-based management is via the farmer-restaurant analogy described in a September 15, 2014, article by Arun Anandasivam on IBM's Thoughts on Cloud site. If you buy your food directly from the farmer, you have a first-hand impression of the person who grew your food, but your options may be limited and you have to do the preparation work. If you buy your food from a restaurant, you likely have a wider selection to choose from and you needn't prepare the meal, but you have less control over the food's path from farm to kitchen, and you have fewer opportunities to determine beforehand whether the food meets your quality requirements.

That's not to say farmers are any more or less trustworthy than restaurants. You use the same senses to ensure you're getting what you paid for, just in different ways. So check out the Morpheus database-as-a-service to see what's on the menu!

TL;DR: Google Analytics stores a massive amount of statistical data from web sites across the globe. Retrieving reports quickly from such a large amount of data requires Google to use a custom solution that is easily scalable whenever more data needs to be stored.

At Google, any number of applications may need to be added to their infrastructure at any time, and each of these could potentially have extremely heavy workloads. Resource demands such as these can be difficult to meet, especially when there is a limited amount of time to get the required updates implemented.

If Google were to use the typical relational database on a single server node, they would need to upgrade their hardware each time capacity is reached. Given the amount of applications being created and data being used by Google, this type of upgrade could quite possibly be necessary on a daily basis!

The load could also be shared across multiple server nodes, but once more than a few additional nodes are required, the complexity of the system becomes extremely difficult to maintain.

With these things in mind, a standard relational database setup would not be a particularly attractive option due to the difficulty of upgrading and maintaining the system on such a large scale.

Finding a Scalable Solution

In order to maintain speed and ensure that such incredibly quick hardware upgrades are not necessary, Google uses its own data storage solution called BigTable. Rather than store data relationally in tables, it stores data as a multi-dimensional sorted map.

This type of implementation falls under a broader heading for data storage, called a key/value store. This method of storage can provide some performance benefits and make the process of scaling much easier.

Information Storage in a Relational Database

Relational databases store each piece of information in a single location, which is typically a column within a table. For a relational database, it is important to normalize the data. This process ensures that there is no duplication of data in other tables or columns.

For example, customer last names should always be stored in a particular column in a particular table. If a customer last name is found in another column or table within the database, then it should be removed and the original column and table should be referenced to retrieve the information.

The downside to this structure is that the database can become quite complex internally. Even a relatively simple query can have a large number of possible paths for execution, and all of these paths must be evaluated at run time to find out which one will be the most optimal. The more complex the database becomes, the more resources will need to be devoted to determining query paths at run time.

Information Storage in a Key/Value Store

With a key/value store, duplicate data is acceptable. The idea is to make use of disk space, which can easily and cost-effectively be upgraded (especially when using a cloud), rather than other hardware resources that are more expensive to bring up to speed.

This data duplication is beneficial when it comes to simplifying queries, since related information can be stored together to avoid having numerous potential paths that a query could take to access the needed data.

Instead of using tables like a relational database, key/value stores use domains. A domain is a storage area where data can be placed, but does not require a predefined schema. Pieces of data within a domain are defined by keys, and these keys can have any number of attributes attached to them.

The attributes can simply be string values, but can also be something even more powerful: data types that match up with those of popular programming languages. These could include arrays, objects, integers, floats, Booleans, and other essential data types used in programming.

With key/value stores, the data integrity and logic are handled by the application code (through the use of one or more APIs) rather than by using a scheme within the database itself. As a result, data retrieval becomes a matter of using the correct programming logic rather than relying on the database optimizer to determine the query path from a large number of possibilities based on the relation it needs to access.

Getting Results

Google needs to store and retrieve copious amounts of data for many applications, included among them are Google Analytics, Google Maps, Gmail, and their popular web index for searching. In addition, more applications and data stores could be added at any time, making their BigTable key/value store an ideal solution for scalability.

BigTable is Google’s own custom solution, so how can a business obtain a similar performance and scalability boost to give its users a better experience? The good news is that there are other key/value store options available, and some can be run as a service from a cloud. This type of service is easily scalable, since more data storage can easily be purchased as needed on the cloud.

A Key/Value Store Option

There are several options for key/value stores. One of these is Mongo, which is designed as an object database that stores information in JSON format. This format is ideal for web applications since JSON data makes it easy to pass data around in a standard format among the various parts of an application that need it.

For example, Mongo is part of the MEAN stack: Mongo, Express, AngularJS, and NodeJS—a popular setup for programmers developing applications. Each of these pieces of the puzzle will send data to and from other one or more of the other pieces. Since everything, including the database, can use the JSON format, passing the data around among the various parts becomes much easier and more standardized.

How to Make Use of Mongo

Mongo can be installed and used on various operating systems, including Windows, Linux, and OS X. In this case, the scalability of the database would need to be maintained by adding storage space to the server on which it is installed.

Another option is to use Mongo as a service on the cloud. This allows for easy scalability, since a request can be made to the service provider to up the necessary storage space at any time. In this way, new applications or additional data storage needs can be handled quickly and efficiently.

Morpheus is a great option for this service. Mongo is offered, as well as a number of other databases. Using Morpheus, a highly scalable database as a service can be running in no time!