The first step in combatting the perpetrators of Internet crimes is to uncover the Darknet in which they operate.

It's getting easier and easier for criminals to infiltrate your company’s network and help themselves to your financial and other sensitive information, and that of your customers. There's a ready market for stolen certificates that make malware look legitimate to antivirus software and other security systems.

The crooks even place orders for stolen account information: One person is shopping for purloined Xbox, GameStop, iTunes, and Target accounts; another is interested only in accounts belonging to Canadian financial institutions. Each stolen record costs from $4 to $10, on average, and customers must buy at least $100 worth of these hijacked accounts. Many of the transactions specify rubles (hint, hint).

Loucif Kharouni, Senior Threat Researcher for security service Damballa, writes in a September 21, 2015, post that the cybercrime economy is thriving on the so-called Darknet, or Dark Web. Criminals now offer cybercrime-as-a-service, allowing anyone with an evil inclination to order up a malware attack, made to order -- no tech experience required.

Criminal sites operate beyond the reach of law enforcement

Sadly, thieves aren't the only criminals profiting from the Darknet. Human traffickers, child pornographers, even murderers are taking advantage of the Internet to commit their heinous crimes, as Dark Reading's Sara Peters reports in a September 16, 2015, article.

Peters cites a report by security firm Bat Blue Networks that claims there are between 200,000 and 400,000 sites on the Darknet. In addition to drug sales and other criminal activities, the sites are home to political dissidents, whistleblowers, and extremists of every description. It's difficult to identify the servers hosting the sites because they are shrouded by virtual private networks and other forms of encryption, according to Bat Blue's researchers.

Most people access the sites using The Onion Router (Tor) anonymizing network. That makes it nearly impossible for law enforcement to identify the criminals operating on the networks, let alone capture and prosecute them. In fact, Bat Blue claims "nation-states" are abetting the criminals, whether knowingly or unknowingly.

The Darknet is populated by everyone from public officials to religious extremists, for as wide a range of purposes. Source: Bat Blue Networks

While hundreds of thousands of sites comprise the Darknet, you won’t find them using the web’s Domain Name System. Instead, the sites communicate by delivering an anonymous service, called a “hidden service,” via updates to the Tor network. Rather than getting a domain from a registrar, the sites authenticate each other by using self-generated public/private key pair addresses.

The public key generates a 16-character hash that ends in .onion to serve as the address that accesses the hidden service. When the connection is established, keys are exchanged to create an encrypted communication channel. In a typical scenario, the user installs a Tor client and web server on a laptop, takes the laptop to a public WiFi access point (avoiding the cameras that are prevalent at many such locations), and uses that connection to register with the Tor network.

The Tor Project explains the six-step procedure for using a hidden service to link anonymously and securely via the Tor network:

1. Party A builds circuits to select introduction points on the Tor network.
2. A hidden service descriptor containing the public key and summaries of each introduction point, and signed by the private key, is uploaded to a distributed hash table on the network.
3. Party B finds the hidden service’s .onion address and downloads the descriptor from the distributed hash table to establish the protected connection to it.
4. Party B creates an “introduce” message encrypted to the hidden service's public key; the message includes the address of the rendezvous point and the one-time secret. The message is sent to one of the introduction points for delivery to the hidden service. (This step is shown in the image below.)
5. The hidden service decrypts the message, finds the rendezvous address and one-time secret, creates a circuit to the rendezvous point, and sends a rendezvous message that contains another one-time secret.
6. The rendezvous point notifies Party B that the connection has been established, and then Party B and the hidden service pass protected messages back and forth.

In the fourth of the six steps required to establish a protected connection to the Tor network, Party B (Ann) sends an “introduce” message to one of the hidden service’s introduction points created by Party A (Bob). Source: Tor Project

Defeating the Darknet starts by removing its cloak of invisibility

Criminals cannot be allowed to operate unfettered in the dark shadows of the Internet. But you can’t arrest what you can’t spot. That’s why the first step in combatting Darknet crime is to shine a light on it. That’s one of the primary goals of the U.S. Defense Research Projects Agency’s Memex program, which Mark Stockley describes in a February 16, 2015, post on the Sophos Naked Security site.

Memex is intended to support domain-specific searches, as opposed to the broad, general scope of commercial search engines such as Google and Bing. Initially, it targets human trafficking and slavery, but its potential uses extend to the business realm, as Computerworld’s Katherine Noyes reports in a February 13, 2015, article.

For example, a company could use Memex to spot fraud attempts and vet potential partners. However, the ability to search for information that isn’t indexed by Google and other commercial search engines presents companies with a tremendous competitive advantage, according to analysts. After all, knowledge is power, and not just for the crooks running amok on the Darknet.

Find out how a CLI can help your company with app development and deployment 

A CLI can help developers and administrators get apps deployed quickly, in addition to the other uses of CLIs, such as automating tasks for an operating system or running helpful commands that can run a ping, get information on files, or execute file system commands.

What is a CLI?

CLI stands for command-line interface, though you might also hear of this under a different name or acronym, such as character user interface (CUI) or command-line user interface. A CLI allows a user to commands to a program using written text rather than a GUI (Graphical User Interface). For many years, CLIs were the main form of interacting with computers. The introduction of GUI operating systems such as Windows and Macintosh eventually made computer operation easier for beginners, but CLIs have remained as an alternate means of interaction, often employed by more advanced users such as administrators or developers.

Source: HubPages 

How do CLIs help Administrators and Developers?

For administrators and developers, a CLI can be a handy tool for interacting with a particular piece of software. Whether it is an operating system, server, or another system, a CLI can provide you with the ability to provide additional parameters to the commands that you need to run. This sort of flexibility helps when you need to configure a command to run a particular way. For example, getting a directory listing in a CLI for Windows is done by typing “dir” and pressing “Enter”. This works well for short listings, but if that particular directory has a large number of files, you may not be able to see what you needed on the screen. To help with this, you can add the “/p” parameter, e.g., “dir /p”. This time, the CLI will display the number of files that will fit on a single page and allow you to continue as needed until you see what you need.

SaaS with a CLI

As an administrator or developer, you may be considering a SaaS package, and having a CLI to work with would be helpful. Morpheus is a service that offers exactly what you need: SaaS with a CLI for you to help your productivity. With the Morpheus CLI, you can quickly provision instances by typing in the necessary commands. For example, the interaction below will create a Node.js instance on Morpheus with ease: 

 $ morpheus instances add "My node app" node

Configurations:

  1) Single Node (node-4.0.0-single)

Selection: 1

Select a Plan:

  1) Memory: 128MB Storage: 1GB

  2) Memory: 256MB Storage: 3GB

  3) Memory: 512MB Storage: 5GB

  4) Memory: 1GB Storage: 10GB

  5) Memory: 2GB Storage: 20GB

  6) Memory: 4GB Storage: 40GB

  7) Memory: 8GB Storage: 80GB

  8) Memory: 16GB Storage: 160GB

Selection: 1

With Morpheus, you can provision apps and databases in real-time to public, private, and hybrid clouds and spin up databases, apps, environments and more with a few simple clicks. You can use the monitoring service to keep track of overall app uptime and response time, while also tracking the vital statistics for each individual piece of your app. With all of these features, why not give Morpheus a try today? Click here to sign up for a demo now. 

Fantastic Five: Super Cloud-based Management Tools

ManageEngineAlarmsOne

• ·       Combines and groups various alarms based on host, network device, application, database, and other criteria.
• ·       Works with Solarwinds, PRTG, Nagios, HP Site Scope, App Dynamic, Loggly, New Relic, Site24x7, Copper Egg, and other monitoring and SaaS products.
• ·       Lets you respond to alarms generated by the dozens of management systems likely to be present in the typical IT environment, without having to sign into each system separately.
• ·       Consolidates alarms to generate a data model that provides insight into your overall monitoring.
• ·       Automatically categorizes alarms based on source, severity, application, host, network component, database, and other characteristics.
• ·       Alarm Poller uses REST APIs or custom scripts to collect and consolidate alarms in widely distributed systems.

Scalr

• ·       Allows IT departments to orchestrate, automate, and enforce policies that apply to multiple cloud apps and platforms.
• ·       Facilitates enforcing limits on virtual machine types and sizes, network use, regional placement, naming conventions, and other operations.
• ·       Supports use of either Microsoft Active Directory or OpenLDAP to centralize authentication and authorization; lets you implement single sign-on across public and private clouds.
• ·       Role-based access control (RBAC) allows managers to restrict user access to individual features, which simplifies policy management when provisioning.
• ·       Sophisticated cost analytics provide a big-picture view of your cloud spending to enhance budgeting and showback/chargeback; customize notifications to administrators when a budget limit is exceeded.
• ·       Integrates smoothly with existing content-management databases, notifications, security, and audit systems via WebHooks dispatched as standard HTTP JSON requests.

Splunk Light

• ·       Log-analysis tool collects and analyzes data from every kind of app, server, and network.
• ·       Uses the Splunk Search Processing Language to correlate events and identify patterns across diverse data sources.
• ·       Offers customizable dashboards that let you drill down from any chart to the raw events underlying it, as well as links to other dashboards, forms, views, or external websites.
• ·       Alerts can automatically trigger such actions as remedial script execution or posts to an RSS feed.

Fabric

• ·       Python library and command-line tool for managing app development and management via SSH.
• ·       Includes operations for executing local or remote shell commands, prompting users for input, uploading/downloading files, and other functions.
• ·       Pass runtime parameters into tasks using a shell-compatible notation, and specify connection information in your fabfile or at the command line rather than at runtime by setting the global host list, env.hosts.
• ·       Context managers preserve directory and local system states between connections, similar to running the CD command during an SSH session to run multiple commands.
• ·       Simpler and quicker to learn than the more-powerful Ansible app-management tool.

Crowbar

• ·       Orchestration framework that features a service-oriented model based on iterative directed convergence with system-wide cross-node dependencies.
• ·       Intended for configuring physical infrastructure comprised of heterogeneous hardware, operating systems, and management tools.
• ·       BIOS settings and RAID configuration set automatically depending on how the system will be used.
• ·       Support for IPMI and WSMan for out-of-band configuration and management.
• ·       Commercial support via RackN, a company founded by Crowbar creators Rob Hershfeld, Greg Althaus, Scott Jensen, and Dan Choquette.

Curious how MongoDB got its name? Here's your quick history lesson for the day. 

Example of a MongoDB query. Source: MongoDB.

The company behind MongoDB

MongoDB was originally developed by MongoDB, Inc., which at the time (2007) was named 10gen. The company was founded by former DoubleClick founders and engineers, specifically Dwight Merriman, Kevin P. Ryan, and Eliot Horowitz.

At first, 10gen wanted to build an open-source platform as a service. The company wanted all of the components of its software to be completely open-source, but could not find a database that met their needs and provided the type of scalability needed for the applications they were building.

The platform 10gen was working on was named Babble and was going to be similar to the Google App Engine. As it turned out, there wasn't a big market for Babble, but both users and non-users of Babble agreed that the database 10gen had created to accompany the platform was excellent and would be happy to use it on its own.

While originally simply dubbed "p", the database was officially named MongoDB, with "Mongo" being short for the word humongous. Given the input 10gen had received about MongoDB, the company decided it would indeed be best to scrap the Babble project and release MongoDB on its own as an open-source database platform in 2009.

By 2012, 10gen had been named number nine on "The Next Big Thing 2012" published by the Wall Street Journal and had 6 offices located in various parts of the world. In 2013, 10gen renamed itself to MongoDB, Inc., wanting to make the strong association with its popular primary product.

The impact of MongoDB

As time went on, MongoDB moved up the ranks to become the most popular type of database for document stores, and the fourth most popular database system overall. It is used by other highly successful companies like eBay, Abobe, LinkedIn, Foursquare, McAfee, Shutterfly, and others.

It is also used by software developers as part of the MEAN stack, which includes MongoDB (database), Express (web app framework), AngularJS (MVC JavaScript front-end framework) and NodeJS (platform for server-side apps). Part of the popularity of this stack is that JavaScript and/or JSON/BSON notation can be used across all members of the stack, allowing developers to easily move through and develop within each piece of the stack.

The MEAN stack. Source: modernweb.

All in all, MongoDB can be an excellent choice for a database for your applications, especially if you deal with large amounts of data that will continually expand over time!

To see how Morpheus can help you get more out of your MongoDB sign up for a demo today!

By Nermin Hajdarbegovic, Technical Editor at Toptal

At a glance, the hosting industry may not appear exciting, but it's grunts in data centres the world over that keep our industry going. They are, quite literally, the backbone of the Internet, and as such they make everything possible: from e-commerce sites, to smart mobile apps for our latest toys. The heavy lifting is done in boring data centres, not on our flashy smartphones and wafer thin notebooks. 

Whether you’re creating a virtual storefront, deploying an app, or simply doing some third-party testing and development, chances are you need some server muscle. The good news is that there is a lot to choose from. The hosting industry may not be loud or exciting, but it never sleeps; it’s a dog eat dog world, with cutthroat pricing, a lot of innovation behind the scenes, and cyclical hardware updates. Cloud, IaaS and PaaS have changed the way many developers and businesses operate, and these are relatively recent innovations.

In this post I will look at some hosting basics from the perspective of a freelance developer: what to choose and what to stay away from. Why did I underline freelance software engineers? Well, because many need their own dev environment, while at the same time working with various clients. Unfortunately, this also means that they usually have no say when it comes to deployment. For example, it’s the client’s decision how and where a particular web app will be hosted, and a freelancer hired on short-term basis usually has no say in the decision. This is a management issue, so I will not address it in this post other than to say that even freelancers need to be aware of options out there. Their hands may be tied, but in some cases clients will ask for their input and software engineers should help them make an informed decision. Earlier this week, we covered one way of blurring the line between development and operations: DevOps. In case you missed that post, I urge you to check it out and see why DevOps integration can have an impact on hosting as well.

Luckily, the hosting industry tries to cater to dev demand, so many of hosting companies offer plans tailored for developers. But wait, aren’t all webhosting plans just as good for developers as these “developer” plans? Is this just clever marketing and a cheap SEO trick?

Filtering Out the Noise 

So, how does one go about finding the right hosting plan? Google is the obvious place to start, so I tried searching for “hosting for developers.” By now, you can probably see where I am going with this. That particular search yielded 85 million results and enough ads to make Google shareholders pop open a bottle of champagne.

If you’re a software engineer looking for good hosting, it’s not a good idea to google for answers. Here’s why.

There is a very good reason for this, and I reached out to some hosting specialists to get a better idea of what goes on behind the scenes.

Adam Wood, Web Hosting Expert and Author of Ultimate Guide to Web Hosting explained: 

“Stay away from Googling ‘hosting for developers.’ That shows you hosts that have spent a lot of money on SEO, not a lot of energy on building an excellent platform.” 

Wood confirmed what most of us knew already: A lot of “hosting for developers” plans are marketing gimmicks. However, he stressed that they often offer perfectly fine hosting plans in their own right.

“The ‘hosting’ is real, the ‘for developers’ part is just marketing,” he added.

Although Wood works for hosting review site WhoIsHostingThis, he believes developers searching for a new host should rely on more than online searches.

Instead of resorting to Google, your best bet for finding the perfect plan for your dev needs is word of mouth and old-fashioned research:

• Check out major tech blogs from developers using the same stack as you.
• Reach out to the community and ask for advice.
• Take a closer look at hosting plans offered by your current host. Look for rapid deployment tools, integration to other developer tools, testing support and so on.
• Make sure you have clear needs and priorities; there’s no room for ambiguity.
• Base your decision on up-to-date information.

Small Hosts May Have Trouble Keeping Up

But what about the hundreds of thousands of hosting plans tailored for developers? Well, they’re really not special and in most cases you can get a similar level of service and support on a “plain Jane” hosting plan.

Is there even a need for these small and inexpensive plans? Yes, there is. Although seasoned veterans probably won’t use them, they are still a piece of the puzzle, allowing small developers, hobbyists and students to hone their skills on cheap, using shared hosting plans that cost less than a gym membership. Nobody is going to host a few local hobby sites on AWS, and kids designing their first WordPress sites won’t get a VPS. In most cases, they will use the cheapest option out there.

Cheap, shared hosting plans are the bread and butter of many hosting outfits, so you can get one from an industry leader, or a tiny, regional host. The trouble with small hosts is that most of them rely on conventional reseller hosting or re-packaging cloud hosting from AWS and other cloud giants. These plans are then marketed as shared hosting plans, VPS plans, or reseller plans.

Bottom line: If something goes wrong with your small reseller plan, who are you going to call in the middle of the night?

Small hosts are fading and this is more or less an irreversible trend. Data centres are insanely capital-intensive; they’re the Internet equivalent of power stations, they keep getting bigger and more efficient, while at the same time competing to offer lower pricing and superior service. This obviously involves a lot of investment, from huge facilities with excellent on-site security and support through air-conditioning, redundant power supply and amazingly expensive Internet infrastructure. On top of that, hosts need a steady stream of cutting edge hardware. Flagship Xeons and SAS SSDs don’t come cheap.

There is simply no room for small players in the data centre game.

Small resellers still have a role to play, usually by offering niche services or a localisation, including local support in various languages not supported by the big host. However, most of these niches and potential advantages don’t mean a whole lot for the average developer.

The PaaS Revolution

Less than a decade ago, the industry revolved around dedicated and shared hosting, and I don’t think I need explain what they are and how they work.

Cloud services entered the fray a few years ago, offering unprecedented reliability and scalability. The latest industry trends offer a number of exciting possibilities for developers in the form of developer-centric Platform-as-a-Service (PaaS) offerings.

PaaS is the new black for many developers. How does it compare to traditional hosting?

Most developers are already familiar with big PaaS services like Heroku, Pantheon and OpenShift. Many of these providers began life as platforms for a specific framework or application. For example, Heroku was a Ruby-on-Rails host, while Pantheon was a Drupal managed-hosting provider, which expanded to WordPress.

PaaS services can be viewed as the next logical step in the evolution of managed hosting. However, unlike managed hosting, PaaS is geared almost exclusively toward developers. This means PaaS services are tailored to meet the needs of individual developers and teams. It’s not simply about hosting; PaaS is all about integrating into a team’s preferred workflow by incorporating a number of features designed to boost productivity. PaaS providers usually offer a host of useful features:

·       Ability to work with other developer tools like GitHub.

·       Supports Continuous Integration (CI) tools like Drone.io, Jenkins, and Travis CI.

·       Allows the creation of multiple, clonable environments for development, testing, beta, and production.

·       Supports various automated testing suites.

Best of all, many PaaS providers offer free developer accounts. Heroku and Pantheon both allow developers to sample the platform, thus encouraging them to use it for projects later on. In addition, if one of these experimental projects takes off, developers are likely to remain on the platform. 

It’s clever marketing, and it’s also an offer a lot of developers can’t afford to ignore. PaaS is here to stay and if you haven’t taken the plunge yet, perhaps it is time to do a little research and see what’s out there.

Traditional Hosting And Cloud Offerings

Dedicated and shared hosting aren’t going anywhere. They were the mainstays of web hosting for two decades and they’re still going strong. A lot of businesses rely on dedicated servers or VPS servers for their everyday operations. Some businesses choose to use cloud or PaaS for specific tasks, alongside their existing server infrastructure.

In some situations, PaaS can prove prohibitively expensive, but powerful dedicated servers don’t come cheap, either. The good news is that PaaS can give you a good idea of the sort of resources you will need before you decide to move to a dedicated server. Further, PaaS services tend to offer better support than managed VPS servers or dedicated servers.

Of course, all this is subjective and depends on your requirements and budget.

PaaS, dedicated servers, VPS plans, or your own slice of the Cloud. What should a freelance software engineer choose?

Call me old-fashioned, but I still believe dedicated servers are the best way of hosting most stuff. However, this only applies to mature projects; development is a whole other ball game. Managed dedicated servers offer exceptional reliability and good levels of support, along with good value for money.

Properly used, dedicated servers and PaaS can speed up deployment as well, as Adam Wood explains:

“I can spin up a new Ruby-on-Rails app on Heroku in a matter of minutes. Doing the same thing on AWS takes me a half a day, and I constantly feel like I’m about to break something.”

Cloud services are inherently more efficient than dedicated hardware because you only use the resources you need at any given time. For example, if you are operating a service that gets most of its traffic during office hours (from users in the Americas), your dedicated server will be underutilised for 12 to 16 hours. Despite this obvious efficiency gap, dedicated servers can still end up cheaper than cloud solutions. In addition, customers can customise and upgrade them the way they see fit.

Cloud is catching up, but dedicated servers will still be around for years to come. They’re obviously not a good solution for individual developers, but are for a lot of businesses. VPS plans cost a lot less than dedicated servers and are easily within the reach of individual developers, even though they don’t offer the same level of freedom as dedicated servers.

What Does This Mean For Freelancers?

The good news is that most freelance software engineers don’t need to worry about every hosting option out there. While it’s true that different clients have different ways of doing things, in most cases it’s the client’s problem rather than yours.

This does not mean that different hosting choices have no implications on freelancers; they do, but they are limited. It is always a good idea to familiarise yourself with the infrastructure before getting on board a project, but there is not much to worry about. Most new hosting services were developed to make developers’ lives easier and keep them focused on their side of the project. One of the positive side-effects on PaaS and cloud adoption is increasing standardisation; most stacks are mature and enjoy wide adoption, so there’s not a lot that can go wrong.

Besides, you can’t do anything about the client’s choice of infrastructure, for better or for worse. But what about your own server environment?

There is no one-size-fits-all solution; it all depends on your requirements, your stack, and your budget. PaaS services are gaining popularity, but they might not be a great solution for developers on a tight budget, or those who don’t need a hosting environment every day. For many freelancers and small, independent developers, VPS is still the way to go. Depending on what you do, an entry-level managed dedicated server is an option, and if you do small turnkey web projects, you may even consider some reseller packages. 

The fact that big hosting companies continue to compete for developers’ business is, ultimately, a good thing. It means they’re forced to roll out timely updates and offer better support across all hosting packages in order to remain competitive. They are not really competing with PaaS and cloud services, but they still want a slice of the pie.

Remember how PaaS providers offer developers various incentives to get on board, just so they could get their business in the long run? It could be argued that conventional hosting companies are trying to do the same by luring novice developers to their platform, hoping that they will be loyal customers and use their servers to host a couple of dozen projects a few years down the road.

The Future Of Hosting

Although the hosting industry may not appear as vibrant and innovative as other tech sectors, this is not entirely fair. Of course, it will always look bland and unexciting compared to some fast-paced sectors, but we’re talking about infrastructure, not some sort of get rich quick scheme.

The hosting industry is changing, and it is innovative. It just takes a bit longer to deploy new technology, that’s all. For example, a logistics company probably changes its company smartphones every year or two, but its delivery vehicles aren’t updated nearly as often, yet they’re the backbone of the business.

Let’s take a quick look at some hosting industry trends that are becoming relevant from a software development perspective:

·       Continual development and growth of Cloud and PaaS services.

·       Evolution of managed hosting into quasi-PaaS services.

·       Increasing integration with industry standard tools.

·       New hardware might make dedicated servers cheaper.

Cloud and PaaS services will continue to mature and grow. More importantly, as competition heats up, prices should come down. The possibility of integrating various development tools and features into affordable hosting plans will continue to make them attractive from a financial perspective. Moving up on the price scale, managed hosting could also evolve to encompass some features and services offered by PaaS. If you’re interested in hosting industry trends, I suggest you check out this Forbes compilation of cloud market forecasts for 2015 and beyond.

Dedicated servers will never be cheap, at least not compared to shared and VPS plans. However, they are getting cheaper, and they could get a boost in the form of frugal and inexpensive ARM hardware. ARM-based processors tend to offer superior efficiency compared to x86 processors, yet they are relatively cheap to develop and deploy. Some flagship smartphones ship with quad-core chips, based on 64-bit Cortex-A57 CPU cores, and the same cores are coming to ARM-based server processors.

As a chip geek, I could go on, but we intend to take an in-depth look at the emerging field of ARM servers in one of our upcoming blog posts, so if you’re interested, stay tuned.

This article originally appeared in Toptal link at https://www.toptal.com/it/hosting-for-freelance-developers-paas 

To try out Morpheus' leading PaaS offering sign up for a free demo here. 

Data logging is one of the most important aspects of most IT pros. So, do you know what it is?

Data logging is often talked about as a helpful tool that you can use when trying to maintain your various servers, databases, and other systems that go into an application. So, what is data logging and what does it do that helps you maintain your applications more easily?

Data Logging Defined Generally speaking, data logging is the recording of data over a period of time by a computer system or a special standalone device which can be tailored to a specific use case. The recorded data can then be retrieved and analyzed to help determine if things ran smoothly during the time the data was being recording, and to help identify what happened if there were any issues that would be in need or further attention. Standalone data loggers are used in many familiar environments to gather information such as weather conditions, traffic conditions, wildlife research, and many others. These devices make it possible for the recording of data to take place 24/7 and automatically, without the need for a person to be present with the data logger. 

A data logger for a weather station. Source: Wikipedia.

For instance, when performing wildlife research, it can be beneficial to have such automated logging, as wildlife may behave differently when one or more humans are present. For the purposes of application monitoring, data logging records information pertinent to the maintenance of the infrastructure that is required for an application to run.

How Data Logging Helps With App Maintenance When maintaining apps, it is always helpful to know when and where something went wrong. In many cases, such logging can help you avoid problems by alerting you that an issue may arise soon (a server beginning to respond slowly, for instance). Data logging can also help you keep track of statistics over time, such as the overall uptime, the uptime of specific servers, average response time, and other data that can help you tweak your applications for optimum uptime and performance.

Morpheus and Monitoring

If you are looking for a monitoring system with excellent data logging and analysis reports, you should give Morpheus a try. With Morpheus, data logging is automatic as you provision servers and apps. Using the available tools, you can monitor the various parts of your system to keep track of uptime, response time, and to be alerted if an issue does arise.

The Morpheus interface is clean and easy to use. Source: Morpheus.

Morpheus also allows you to provision apps in a single click and provides ease of use for developers with APIs and a CLI. In addition, backups are also automatic, and you can have redundancy as needed to avoid potentially long waits for disaster recovery to take place. Sign up for a demo and we'll let you try out Morpheus for free today. 

If you need to use pagination in your Redis app, there are a couple of strategies you can use to achieve the necessary functionality. While pagination can be challenging, a quick overview of each of these techniques should be helpful in making your job of choosing a method and implementing it a little easier. There are several strategies for pagination in Redis. Find out what they are and the pros and cons of each!

In Redis, you have a couple of options from which to choose. You can use the SSCAN command or you can use sorted sets. Each of these has their own advantages, so choose the one that works best for your application and its infrastructure.

Using the SSCAN Command

The SSCAN command is part of a group of commands similar to the regular SCAN command. These include:

• SCAN - Used to iterate over the set of keys in the current database.
• SSCAN - Used to iterate over elements of sets.
• HSCAN - Used to iterate fields hashes and associated values.
• ZSCAN - Used to iterate elements of sorted sets and their scores.

Example of scan iteration. Source: Redis.

So, while the regular SCAN command iterates over the database keys, the SSCAN command can iterate over elements of sets. By using the returned SSCAN cursor, you could paginate over a Redis set.

The downside is that you need some way to persist the value of the cursor, and if there are concurrent users this could lead to some odd behavior, since the cursor may not be where it is expected. However, this can be useful for applications where traffic to these paginated areas may be lighter.

Using Sorted Sets

In Redis, sorted sets are a non-repeating collection of strings associated with a score. This score is used to order the set from the smallest to the largest score. This data type allows for fast updating by giving you easy access to elements, even if the elements are in the middle of the set.

An example of sorted set elements Source: Redis.

To paginate, you can use the ZRANGE command to select a range of elements in a sorted set based on their scores. So, you could, for example, select scores from 1-20, 21-40, and so on. By programmatically adjusting the range as the user moves through the data, you can achieve the pagination you need for your application.

Since sorted sets and ZRANGE do this task more intuitively than using a scan, it is often the preferred method of pagination, and is easier to implement with multiple users, since you can programmatically keep track of which ZRANGE each user is selecting at any given time.

In the end, you can choose which method works for your particular situation. If you have a smaller application with less traffic, a scan may work for you. If; however, you need a more robust solution for larger data sets or more highly utilized applications, it may be best to go ahead and use ZRANGE with sorted sets to achieve pagination in your application.

At times, a web app or web site may appear to be down when the server it is on appears to be functioning properly. When this happens, it is important to know where the issue resides, as it may be easy to fix, or may require a lot of work or contacting others. One of the possibilities when a site is in this state is whether or not the DNS server is up to date and pointing others to the proper server in order to load your site or app.

What is DNS?
DNS stands for Domain Name System. It is the tool that allows a typical URL, such as http://gomorpheus.com , to point to the server on which the actual web site or app resides. Once a computer finds the DNS information it needs for mapping a base URL to a server address, it will remember it for a period of time, until its TTL (Time To Live) has been reached.

How DNS can contribute to downtime

DNS can contribute to downtime in several ways:

1. The DNS server has the wrong information stored about the server to which the domain should be pointed. For example, the server is actually at the IP address 204.268.130.100, but the DNS entry has the server at 204.268.120.100. Here, changing the entry to the proper address will fix the situation.
2. The DNS server is down. In such a case, computers that do not have the DNS information cached cannot reach the DNS server to look up the proper address. This will require getting your DNS server back up and running, or contacting the proper people to do this if it is not your server.
3. The changes haven’t propagated and updated caches yet. Since computers cache DNS information in the operating system and browser, this could be the case.

If the user is affected by number three above, there are a couple of things to try:

1. Have the user close the web browser, reopen it, and try again. Browsers have a tendency to cache DNS information, so this may solve the issue.
2. Have the user clear the DNS cache on their operating system. This can be done from a shell, for example, the commands to do this in Windows and OSX are shown below:

#Windows:

ipconfig /flushdns

#OSX:
sudo killall -HUP mDNSResponder

Examples of clearing the DNS cache

Monitoring with Morpheus

Do you want to be notified when your site or app is having issues? If you are looking for a monitoring system with excellent data logging and analysis reports, you should give Morpheus a try. With Morpheus, data logging is automatic as you provision servers and apps. Using the available tools, you can monitor the various parts of your system to keep track of uptime, response time, and to be alerted if an issue does arise.

The Morpheus interface is clean and easy to use. 

Morpheus allows you to provision apps in a single click, and provides ease of use for developers with APIs and a CLI. In addition, backups are also automatic, and you can have redundancy as needed to avoid potentially long waits for disaster recovery to take place. So, why not register an account or try out Morpheus for free today?

By Gergely Kalman, Security Specialist at Toptal

For all too many companies, it’s not until after a breach has occurred that web security becomes a priority. During my years working as an IT Security professional, I have seen time and time again how obscure the world of IT Security is to so many of my fellow programmers.

An effective approach to IT security must, by definition, be proactive and defensive. Toward that end, this post is aimed at sparking a security mindset, hopefully injecting the reader with a healthy dose of paranoia.

In particular, this guide focuses on 10 common and significant web security pitfalls to be aware of, including recommendations on how they can be avoided. The focus is on the Top 10 Web Vulnerabilities identified by the Open Web Application Security Project (OWASP), an international, non-profit organization whose goal is to improve software security across the globe.

A little web security primer before we start – authentication and authorization 

When speaking with other programmers and IT professionals, I often encounter confusion regarding the distinction between authorization and authentication. And of course, the fact the abbreviation auth is often used for both helps aggravate this common confusion. This confusion is so common that maybe this issue should be included in this post as “Common Web Vulnerability Zero”.

So before we proceed, let’s clearly the distinction between these two terms:

• Authentication: Verifying that a person is (or at least appears to be) a specific user, since he/she has correctly provided their security credentials (password, answers to security questions, fingerprint scan, etc.).
• Authorization: Confirming that a particular user has access to a specific resource or is granted permission to perform a particular action.

Stated another way, authentication is knowing who an entity is, while authorization is knowing what a given entity can do. 

Common Mistake #1: Injection flaws

Injection flaws result from a classic failure to filter untrusted input. It can happen when you pass unfiltered data to the SQL server (SQL injection), to the browser (XSS – we’ll talk about this later), to the LDAP server (LDAP injection), or anywhere else. The problem here is that the attacker can inject commands to these entities, resulting in loss of data and hijacking clients’ browsers. 

Anything that your application receives from untrusted sources must be filtered, preferably according to a whitelist. You should almost never use a blacklist, as getting that right is very hard and usually easy to bypass. Antivirus software products typically provide stellar examples of failing blacklists. Pattern matching does not work. 

Prevention: The good news is that protecting against injection is “simply” a matter of filtering your input properly and thinking about whether an input can be trusted. But the bad news is that all input needs to be properly filtered, unless it can unquestionably be trusted (but the saying “never say never” does come to mind here).

In a system with 1,000 inputs, for example, successfully filtering 999 of them is not sufficient, as this still leaves one field that can serve as the Achilles heal to bring down your system. And you might think that putting an SQL query result into another query is a good idea, as the database is trusted, but if the perimeter is not, the input comes indirectly from guys with malintent. This is called Second Order SQL Injection in case you’re interested.

Since filtering is pretty hard to do right (like crypto), what I usually advise is to rely on your framework’s filtering functions: they are proven to work and are thoroughly scrutinized. If you do not use frameworks, you really need to think hard about whether not using them really makes sense in your environment. 99% of the time it does not.

Common Mistake #2: Broken Authentication

This is a collection of multiple problems that might occur during broken authentication, but they don’t all stem from the same root cause.

Assuming that anyone still wants to roll their own authentication code in 2014 (what are you thinking??), I advise against it. It is extremely hard to get right, and there are a myriad of possible pitfalls, just to mention a few:

1. The URL might contain the session id and leak it in the referer header to someone else.
2. The passwords might not be encrypted either in storage or transit.
3. The session ids might be predictable, thus gaining access is trivial.
4. Session fixation might be possible.
5. Session hijacking might be possible, timeouts not implemented right or using HTTP (no SSL), etc…

Prevention: The most straightforward way to avoid this web security vulnerability is to use a framework. You might be able to implement this correctly, but the former is much easier. In case you do want to roll your own code, be extremely paranoid and educate yourself on what the pitfalls are. There are quite a few.

Common Mistake #3: Cross Site Scripting (XSS)

This is a fairly widespread input sanitization failure (essentially a special case of common mistake #1). An attacker gives your web application JavaScript tags on input. When this input is returned to the user unsanitized, the user’s browser will execute it. It can be as simple as crafting a link and persuading a user to click it, or it can be something much more sinister. On page load the script runs and, for example, can be used to post your cookies to the attacker.

Prevention: There’s a simple web security solution: don’t return HTML tags to the client. This has the added benefit of defending against HTML injection, a similar attack whereby the attacker injects plain HTML content (such as images or loud invisible flash players) – not high-impact but surely annoying (“please make it stop!”). Usually, the workaround is simply converting all HTML entities, so that script is returned as <script>. The other often employed method of sanitization is using regular expressions to strip away HTML tags using regular expressions on < and > , but this is dangerous as a lot of browsers will interpret severely broken HTML just fine. Better to convert all characters to their escaped counterparts. 

Common Mistake #4: Insecure Direct Object References

This is a classic case of trusting user input and paying the price in a resulting security vulnerability. A direct object reference means that an internal object such as a file or database key is exposed to the user. The problem with this is that the attacker can provide this reference and, if authorization is either not enforced (or is broken), the attacker can access or do things that they should be precluded from.

For example, the code has a download.php module that reads and lets the user download files, using a CGI parameter to specify the file name (e.g.download.php?file=something.txt). Either by mistake or due to laziness, the developer omitted authorization from the code. The attacker can now use this to download any system files that the user running PHP has access to, like the application code itself or other data left lying around on the server, like backups. Uh-oh.

Another common vulnerability example is a password reset function that relies on user input to determine whose password we’re resetting. After clicking the valid URL, an attacker can just modify the usernamefield in the URL to say something like “admin”.

Incidentally, both of these examples are things I myself have seen appearing often “in the wild”.

Prevention: Perform user authorization properly and consistently, and whitelist the choices. More often than not though, the whole problem can be avoided by storing data internally and not relying on it being passed from the client via CGI parameters. Session variables in most frameworks are well suited for this purpose.

Common Mistake #5: Security misconfiguration

In my experience, web servers and applications that have been misconfigured are way more common than those that have been configured properly. Perhaps this because there is no shortage of ways to screw up. Some examples:

1. Running the application with debug enabled in production.
2. Having directory listing enabled on the server, which leaks valuable information.
3. Running outdated software (think WordPress plugins, old PhpMyAdmin).
4. Having unnecessary services running on the machine.
5. Not changing default keys and passwords. (Happens way more frequently than you’d believe!)
6. Revealing error handling information to the attackers, such as stack traces.

Prevention: Have a good (preferably automated) “build and deploy” process, which can run tests on deploy. The poor man’s security misconfiguration solution is post-commit hooks, to prevent the code from going out with default passwords and/or development stuff built in.

Common Mistake #6: Sensitive data exposure

This web security vulnerability is about crypto and resource protection. Sensitive data should be encrypted at all times, including in transit and at rest. No exceptions. Credit card information and user passwords should never travel or be stored unencrypted, and passwords should always be hashed. Obviously the crypto/hashing algorithm must not be a weak one – when in doubt, use AES (256 bits and up) and RSA (2048 bits and up).

And while it goes without saying that session IDs and sensitive data should not be traveling in the URLs and sensitive cookies should have the secure flag on, this is very important and cannot be over-emphasized.

Prevention:

• In transit: Use HTTPS with a proper certificate and PFS (Perfect Forward Secrecy). Do not accept anything over non-HTTPS connections. Have the secure flag on cookies.
• In storage: This is harder. First and foremost, you need to lower your exposure. If you don’t need sensitive data, shred it. Data you don’t have can’t be stolen. Do not store credit card information ever, as you probably don’t want to have to deal with being PCI compliant. Sign up with a payment processor such asStripe or Braintree. Second, if you have sensitive data that you actually do need, store it encrypted and make sure all passwords are hashed. For hashing, use of bcrypt is recommended. If you don’t use bcrypt, educate yourself on salting and rainbow tables.

And at the risk of stating the obvious, do not store the encryption keys next to the protected data. That’s like storing your bike with a lock that has the key in it. Protect your backups with encryption and keep your keys very private. And of course, don’t lose the keys!

Common Mistake #7: Missing function level access control

This is simply an authorization failure. It means that when a function is called on the server, proper authorization was not performed. A lot of times, developers rely on the fact that the server side generated the UI and they think that the functionality that is not supplied by the server cannot be accessed by the client. It is not as simple as that, as an attacker can always forge requests to the “hidden” functionality and will not be deterred by the fact that the UI doesn’t make this functionality easily accessible. Imagine there’s an /adminpanel, and the button is only present in the UI if the user is actually an admin. Nothing keeps an attacker from discovering this functionality and misusing it if authorization is missing.

Prevention: On the server side, authorization must always be done. Yes, always. No exceptions or vulnerabilities will result in serious problems.

Common Mistake #8: Cross Site Request Forgery (CSRF)

This is a nice example of a confused deputy attack whereby the browser is fooled by some other party into misusing its authority. A 3rd party site, for example, can make the user’s browser misuse it’s authority to do something for the attacker.

In the case of CSRF, a 3rd party site issues requests to the target site (e.g., your bank) using your browser with your cookies / session. If you are logged in on one tab on your bank’s homepage, for example, and they are vulnerable to this attack, another tab can make your browser misuse its credentials on the attacker’s behalf, resulting in the confused deputy problem. The deputy is the browser that misuses its authority (session cookies) to do something the attacker instructs it to do.

Consider this example:

Attacker Alice wants to lighten target Todd’s wallet by transfering some of his money to her. Todd’s bank is vulnerable to CSRF. To send money, Todd has to access the following URL:

http://example.com/app/transferFunds?amount=1500&destinationAccount=4673243243

After this URL is opened, a success page is presented to Todd, and the transfer is done. Alice also knows, that Todd frequently visits a site under her control at blog.aliceisawesome.com, where she places the following snippet:

img src="http://example.com/app/transferFunds?amount=1500&destinationAccount=4673243243" width="0" height="0" />

Upon visiting Alice’s website, Todd’s browser thinks that Alice links to an image, and automatically issues an HTTP GET request to fetch the “picture”, but this actually instructs Todd’s bank to transfer $1500 to Alice.

Incidentally, in addition to demonstrating the CSRF vulnerability, this example also demonstrates altering the server state with an idempotent HTTP GET request which is itself a serious vulnerability. HTTP GET requestsmust be idempotent (safe), meaning that they cannot alter the resource which is accessed. Never, ever, ever use idempotent methods to change the server state.

Fun fact: CSRF is also the method people used for cookie-stuffing in the past until affiliates got wiser.

Prevention: Store a secret token in a hidden form field which is inaccessible from the 3rd party site. You of course always have to verify this hidden field. Some sites ask for your password as well when modifying sensitive settings (like your password reminder email, for example), although I’d suspect this is there to prevent the misuse of your abandoned sessions (in an internet cafe for example).

Common Mistake #9: Using components with known vulnerabilities

The title says it all. I’d again classify this as more of a maintenance/deployment issue. Before incorporating new code, do some research, possibly some auditing. Using code that you got from a random person onGitHub or some forum might be very convenient, but is not without risk of serious web security vulnerability.

I have seen many instances, for example, where sites got owned (i.e., where an outsider gains administrative access to a system), not because the programmers were stupid, but because a 3rd party software remained unpatched for years in production. This is happening all the time with WordPress plugins for example. If you think they will not find your hidden phpmyadmininstallation, let me introduce you to dirbuster.

The lesson here is that software development does not end when the application is deployed. There has to be documentation, tests, and plans on how to maintain and keep it updated, especially if it contains 3rd party or open source components.

Prevention:

• Exercise caution. Beyond obviously using caution when using such components, do not be a copy-paste coder. Carefully inspect the piece of code you are about to put into your software, as it might be broken beyond repair (or in some cases, intentionally malicious).
• Stay up-to-date. Make sure you are using the latest versions of everything that you trust, and have a plan to update them regularly. At least subscribe to a newsletter of new security vulnerabilities regarding the product.

Common Mistake #10: Unvalidated redirects and forwards

This is once again an input filtering issue. Suppose that the target site has a redirect.php module that takes a URL as a GETparameter. Manipulating the parameter can create a URL on targetsite.comthat redirects the browser to malwareinstall.com. When the user sees the link, they will see targetsite.com/blahblahblahwhich the user thinks is trusted and is safe to click. Little do they know that this will actually transfer them onto a malware drop (or any other malicious) page. Alternatively, the attacker might redirect the browser to targetsite.com/deleteprofile?confirm=1.

It is worth mentioning, that stuffing unsanitized user-defined input into an HTTP header might lead to header injection which is pretty bad.

Prevention: Options include:

• Don’t do redirects at all (they are seldom necessary).
• Have a static list of valid locations to redirect to.
• Whitelist the user-defined parameter, but this can be tricky.

Epilogue

I hope that I have managed to tickle your brain a little bit with this post and to introduce a healthy dose of paranoia and web security vulnerability awareness.

The core takeaway here is that age-old software practices exist for a reason and what applied back in the day for buffer overflows, still apply for pickled strings in Python today. Security helps you write correct(er) programs, which all programmers should aspire to.

Please use this knowledge responsibly, and don’t test pages without permission!

For more information and more specific attacks, have a look at:https://www.owasp.org/index.php/Category:Attack.

This post originally appeared in the Toptal blog: https://www.toptal.com/security/10-most-common-web-security-vulnerabilities

 To see how Morpheus can help you get more out of your MongoDB sign up for a demo today!

Note: When we recently launched, we were thrilled to have SQL Guru Pinal Dave give Morpheus a spin. It turns out that he had a great experience, and as he is indeed an SQLAuthority, we thought we'd share his post here as well. Without further delay, Pinal shares his thoughts below:

I love my weekend projects. Everybody does different activities in their weekend – like traveling, reading or just nothing. Every weekend I try to do something creative and different in the database world. The goal is I learn something new and if I enjoy my learning experience I share with the world. This weekend, I decided to explore Cloud Database As A Service – Morpheus. In my career I have managed many databases in the cloud and I have good experience in managing them.

I should highlight that today’s applications use multiple databases from SQL for transactions and analytics, NoSQL for documents, In-Memory for caching to Indexing for search. Provisioning and deploying these databases often require extensive expertise and time. Often these databases are also not deployed on the same infrastructure and can create unnecessary latency between the application layer and the databases. Not to mention the different quality of service based on the infrastructure and the service provider where they are deployed.

Moreover, there are additional problems that I have experienced with traditional database setup when hosted in the cloud:

• Database provisioning & orchestration
• Slow speed due to hardware issues
• Poor Monitoring Tools
• High Network Latency

Now if you have a great software and expert network engineer, you can continuously work on above problems and overcome them. However, not every organization have the luxury to have top notch experts in the field. Now above issues are related to infrastructure, but there are a few more problems which are related to software/application as well.

Here are the top three things which can be problems if you do not have application expert: 

• Replication and Clustering
• Simple provisioning of the hard drive space
• Automatic Sharding

Well, Morpheus looks like a product build by experts who have faced similar situation in the past. The product pretty much addresses all the pain points of developers and database administrators.

What is different about Morpheus is that it offers a variety of databases from MySQL, MongoDB, ElasticSearch to Redis as a service. Thus users can pick and chose any combination of these databases. All of them can be provisioned in a matter of minutes with a simple and intuitive point and click user interface. The Morpheus cloud is built on Solid State Drives (SSD) and is designed for high-speed database transactions. Inaddition it offers a direct link to Amazon Web Services to minimize latency between the application layer and the databases.

Here are the few steps on how one can get started with Morpheus. Follow along with me. First go to http://www.gomorpheus.com and register for a new and free account.

Step 1: Signup

It is very simple to signup for Morpheus.

Step 2: Select your database

I use MySQL for my daily routine, so I have selected MySQL. Upon clicking on the big red button to add Instance, it prompted a dialogue of creating a new instance.

Step 3: Create User

Now we just have to create a user in our portal which we will use to connect to a database hosted at Morpheus. Click on your database instance and it will bring you to User Screen. Over here you will notice once again a big red button to create a new user. I created a user with my first name.

Step 4: Configure Your MySQL Client

I used MySQL workbench and connected to MySQL instance, which I had created with an IP address and user.

That’s it! You are connecting to MySQL instance. Now you can create your objects just like you would create on your local box. You will have all the features of the Morpheus when you are working with your database.

Dashboard

While working with Morpheus, I was most impressed with its dashboard. In future blog posts, I will write more about this feature. Also with Morpheus you use the same process for provisioning and connecting with other databases: MongoDB, ElasticSearch and Reddis.

TL;DR: Even experienced SQL programmers can sometimes be thrown for an infinite loop -- or other code failure -- by one of the many pitfalls of the popular development platform. Get the upper hand by monitoring and managing your databases in the cloud via the Morpheus database-as-a-service.

When an database application goes belly up, the cause can often be traced to sloppy coding -- but not always. Every now and then, the reason for a misbehaving app is an idiosyncrasy in the platform itself. Here's how to prevent your database from tripping over one of these common SQL errors.

In an August 11, 2014, article in the Database Journal, Rob Gravelle describes a "gotcha" (not a bug) in the way MySQL handles numeric value overflows. When a value supplied by an automated script or application is outside the range of a column data type, MySQL truncates the value to an entry within the acceptable range.

Generally, a database system will respond to an invalid value by generating an error instructing the script to proceed or abort, or it will substitute the invalid error with its best guess as to which valid entry was intended. Of course, truncating or substituting the entered value is almost certainly going to introduce an error into the table. Gravelle explains how to override MySQL's default handling of overflow conditions to ensure it generates an error, which is the standard response to invalid entries by most other databases.

Don't fall victim to one of the most-common developer mistakes

According to Justin James on the Tech Republic site, the most common database programming no-no is misuse of primary keys. James insists that primary keys should have nothing at all to do with the application data in a row. Except in the "most unusual of circumstances," primary keys should be generated sequentially or randomly by the database upon row insertion and should not be changed. If they're not system values managed by the system, you're likely to encounter problems when you change the underlying data or migrate the data to another system.

Another frequent cause of program problems is overuse of stored procedures, which James describes as a "maintenance disaster." There's no easy way to determine which applications are using a particular stored procedure, so you end up writing a new one when you make a significant change to an app rather than adapting an existing stored procedure. Instead, James recommends that you use advanced object-relational mappers (ORMs).

Not every developer is sold on ORMs, however. On his Experimental Thoughts blog, Jeff Davis explains why he shies away from ORMs. Because ORMs add more lines of code between the application and the data, they invite more semantic errors. Davis points out that debugging in SQL is simpler when you can query the database as if you were an application. The more lines of code between the application error and the database, the more difficult it is to find the glitch.

One of the common database errors identified by Thomas Larock on the SQL Rockstar site is playing it safe by overusing the BIGINT data type. If you're certain no value in a column will exceed 100,000, there's no need to use the 8-byte BIGINT data type when the 4-byte INT data type will suffice. You may not think a mere 4 bytes is significant, but what if the table ends up with 2 million rows? Then your app is wasting 7.8MB of storage. Similarly, if you know you won't need calendar dates before the year 1900 or after 2079, using SMALLDATETIME will make our app much more efficient.

On the SQL Skills site, Kimberly Tripp highlights another common database-design error: use of non-sequential globally unique identifiers, or GUIDs. In addition to creating fragmentation in the base table, non-sequential GUIDs are four times wider than an INT-based identity.

Is your app's failure to launch due to poor typing skills?

Maybe the best way to start your hunt for a coding error is via a printout rather than with a debugger. That's the advice of fpweb.net's tutorial on troubleshooting SQL errors. The problem could be due to a missing single-quote mark, or misuse of double quotes inside a string. If you get the “No value given for one or more required parameters” error message, make sure your column and table names are spelled correctly

Likewise, if the error states “Data type mismatch in criteria expression,” you may have inserted letters or symbols in a column set for numeric values only (or vice-versa). The FromDual site provides a complete list of MySQL error codes and messages, including explanations for many of the codes, possible sources for the errors, and instructions for correcting many of the errors.

The FromDual site explains how to find sources of information about MySQL error messages. 

For many of the MySQL error messages, the FromDual index provides an explanation, reasons for the error message's appearance, and potential fixes.

Cloud database service helps ensure clean, efficient code

One of the benefits of the Morpheus cloud database-as-a-service is the ability to analyze your database in real time to identify and address potential security vulnerabilities and other system errors. TechTarget's Brien Posey points to another benefit of the database-as-a-service model: By building redundancy into all levels of their infrastructure, cloud database services help organizations protect against data loss and ensure high availability.

In addition to auto backups, replication, and archiving, Morpheus's service features a solid-state-disk-backed infrastructure that increases I/O operations per second (IOPs) by 100 times. Latency is further reduced via direct connections to EC2. Databases are monitored and managed continuously by Morpheus's crack team of DBAs and by the service's sophisticated robots.

Morpheus supports MongoDB, MySQL, Redis, and Elasticsearch. Platform support includes Amazon Web Services, Rackspace, Hiroku, Joyent, Cloud Foundry, and Windows Azure. Visit the Morpheus site for pricing information; free databases are available during the service's beta.

[TL:DR] The theft of hundreds of millions of user IDs, passwords, and email addresses was made possible by a database programming technique called dynamic SQL, which makes it easy for hackers to use SQL injection to gain unfettered access to database records. To make matters worse, the dynamic SQL vulnerability can be avoided by using one of several simple programming alternatives.

How is it possible for a simple hacking method which has been publicized for as many as 10 years to be used by Russian cybercriminals to amass a database of more than a billion stolen user IDs and passwords? Actually, the total take by the hackers in the SQL injection attacks revealed earlier this month by Hold Security was 1.2 billion IDs and passwords, along with 500 million email addresses, according to an article written by Nicole Perlroth and David Gelles in the August 5, 2014, New York Times.

Massive data breaches suffered by organizations of all sizes in recent years can be traced to a single easily preventable source, according to security experts. In an interview with IT World Canada's Howard Solomon, security researcher Johannes Ullrich of the SANS Institute blames an outdated SQL programming technique that continues to be used by some database developers. The shocker is that blocking such malware attacks is as easy as using two or three lines of code in place of one. Yes, according to Ullrich, it's that simple.

The source of the vulnerability is dynamic SQL, which allows developers to create dynamic database queries that include user-supplied data. The Open Web Application Security Project (OWASP) identifies SQL, OS, LDAP, and other injection flaws as the number one application security risk facing developers. An injection involves untrusted data being sent to an interpreter as part of a command or query. The attacker's data fools the interpreter into executing commands or accessing data without authentication.

According to OWASP, injections are easy for hackers to implement, difficult to discover via testing (but not by examining code), and potentially severely damaging to businesses.

The OWASP SQL Injection Prevention Cheat Sheet provides a primer on SQL injection and includes examples of unsafe and safe string queries in Java, C# .NET, and other languages.

An example of an unsafe Java string query (top) and a safe Java PreparedStatement (bottom).

Dynamic SQL lets comments be embedded in a SQL statement by setting them off with hyphens. It also lets multiple SQL statements to be strung together, executed in a batch, and used to query metadata from a standard set of system tables, according to Solomon.

Three simple programming approaches to SQL-injection prevention

OWASP describes three techniques that prevent SQL injection attacks. The first is use of prepared statements, which are also referred to as parameterized queries. Developers must first define all the SQL code, and then pass each parameter to the query separately, according to the OWASP's prevention cheat sheet. The database is thus able to distinguish code from data regardless of the user input supplied. A would-be attacker is blocked from changing the original intent of the query by inserting their own SQL commands.

The second prevention method is to use stored procedures. As with prepared statements, developers first define the SQL code and then pass in the parameters separately. Unlike prepared statements, stored procedures are defined and stored in the database itself, and subsequently called from the application. The only caveat to this prevention approach is that the procedures must not contain dynamic SQL, or if it can't be avoided, then input validation or another technique must be employed to ensure no SQL code can be injected into a dynamically created query.

The last of the three SQL-injection defenses described by OWASP is to escape all user-supplied input. This method is appropriate only when neither prepared statements nor stored procedures can be used, whether because doing so would break the application or render its performance unacceptable. Also, escaping all user-supplied input doesn't guarantee your application won't be vulnerable to a SQL injection attack. That's why OWASP recommends it only as a cost-effective way to retrofit legacy code. 

All databases support one or more character escaping schemes for various types of queries. You could use an appropriate escaping scheme to escape all user-supplied input. This prevents the database from mistaking the user-supplied input for the developer's SQL code, which in turn blocks any SQL injection attempt.

The belt-and-suspenders approach to SQL-injection prevention

Rather than relying on only one layer of defense against a SQL injection attack, OWASP recommends a layered approach via reduced privileges and white list input validation. By minimizing the privileges assigned to each database account in the environment, DBAs can reduce the potential damage incurred by a successful SQL injection breach. Read-only accounts should be granted access only to those portions of database tables they require by creating a specific view for that specific level of access. Database accounts rarely need create or delete access, for example. Likewise, you can restrict the stored procedures certain accounts can execute. Most importantly, according to OWASP, minimize the privileges of the operating system account the database runs under. MySQL and other popular database systems are set with system or root privileges by default, which likely grants more privileges than the account requires.

Adopting the database-as-a-service model limits vulnerability

Organizations of all sizes are moving their databases to the cloud and relying on services such as Morpheus to ensure safe, efficient, scalable, and affordable management of their data assets. Morpheus supports MongoDB, MySQL, Redis, ElasticSearch, and other DB engines. The service's real-time monitoring lets you analyze and optimize the performance of database applications.

In addition to 24/7 monitoring of your databases, Morpheus provides automatic backup, restoration, and archiving of your data, which you can access securely via a VPN connection. The databases are stored on Morpheus's solid-state drives for peak performance and reliability.  

TL; DR: The tremendous growth predicted for the open-source Hadoop architecture for data analysis is driven by the mind-boggling increase in the amount of structured and unstructured data in organizations, and the need for sophisticated, accessible tools to extract business and market intelligence from the data. New cloud services such as Morpheus let organizations of all sizes realize the potential of Big Data analysis.

The outlook is rosy for Hadoop -- the open-source framework designed to facilitate distributed processing of huge data sets. Hadoop is increasingly attractive to organizations because it delivers the benefits of Big Data while avoiding infrastructure expenses.

A recent report from Allied Market Research concludes that the Hadoop market will realize a compound annual growth rate of 58.2 percent from 2013 to 2020, to a total value of $50.2 billion in 2020, compared to $1.5 billion in 2012.

Allied Market Research forecasts a $50.2 billion global market for Hadoop services by the year 2020.

Just how "big" is Big Data? According to IBM, 2.5 quintillion bytes of data are created every day, and 90 percent of all the data in the world was created in the last two years. Realizing the value of this huge information store requires data-analysis tools that are sophisticated enough, cheap enough, and easy enough for companies of all sizes to use.

Many organizations continue to consider their proprietary data too important a resource to store and process off premises. However, cloud services now offer security and availability equivalent to that available for in-house systems. By accessing their databases in the cloud, companies also realize the benefits of affordable and scalable cloud architectures.

The Morpheus database-as-a-service offers the security, high availability, and scalability organizations require for their data-intelligence operations. Performance is maximized through Morpheus's use of 100-percent bare-metal SSD hosting. The service offers ultra-low latency to Amazon Web Services and other peering points and cloud hosting platforms.

The Nuts and Bolts of Hadoop for Big Data Analysis

The Hadoop architecture distributes both data storage and processing to all nodes on the network. By placing the small program that processes the data in the node with the much larger data sets, there's no need to stream the data to the processing module. The processor splits its logic between a map and a reduce phase. The Hadoop scheduling and resource management framework executes the map and reduce phases in a cluster environment.

The Hadoop Distributed File System (HDFS) data storage layer uses replicas to overcome node failures and is optimized for sequential reads to support large-scale parallel processing. The market for Hadoop really took off when the framework was extended to support the Amazon Web Services S3 and other cloud-storage file systems.

Adoption of Hadoop in small and midsize organizations has been slow despite the framework's cost and scalability advantages because of the complexity of setting up and running Hadoop clusters. New services do away with much of the complexity by offering Hadoop clusters that are managed and ready to use: there's no need to configure or install any services on the cluster nodes.

Netflix data warehouse combines Hadoop and Amazon S3 for infinite scalability

For its petabyte-scale data warehouse, Netflix chose Amazon's Storage Service (S3) over the Hadoop Distributed File System for the cloud-based service's dynamic scalability and limitless data and computational power. Netflix collects data from billions of streaming events from televisions, computers, and mobile devices.

With S3 as its data warehouse, Hadoop clusters with hundreds of nodes can be configured for various workloads, all able to access the same data. Netflix uses Amazon's Elastic MapReduce distribution of Hadoop and has developed its own Hadoop Platform as a Service, which it calls Genie. Genie lets users submit jobs from Hadoop, Pig, Hive, and other tools without having to provision new clusters or install new clients via RESTful APIs.

The Netflix Hadoop-S3 data warehouse offers unmatched elasticity in terms of data and computing power in a widely distributed network.

There is clearly potential in combining Hadoop and cloud services, as Wired's Marco Visibelli explains in an August 13, 2014, article. Visibelli describes how companies leverage Big Data for forecasting by scaling from small projects via Amazon Web Services and scaling up as their small projects succeed. For example, a European car manufacturer used Hadoop to combine several supplier databases into a single 15TB database, which saved the company $16 million in two years.

Hadoop opens the door to Big Data for organizations of all sizes. Projects that leverage the scalability, security, accessibility, and affordability of cloud services such as Morpheus's database as a service have a much greater chance of success.

TL;DR: As IT managers gain confidence in the reliability and security of cloud services, it becomes more difficult for them to ignore the cloud's many benefits for all their business's operations. Companies have less hardware to purchase and maintain, they spend only for the storage and processing they need, and they can easily monitor and manage their applications. With the Morpheus database as a service you get all of the above running on a high-availability network that features 24-7 support.

Give any IT manager three wishes and they'll probably wish for three fewer things to worry about. How about 1) having less hardware to buy and manage, 2) having to pay for only the storage and processing you need, and 3) being able to monitor and test applications from a single easy-to-use console?

Knowing the built-in cynicism of many data-center pros, they're likely to scoff at your offer, or at least suspect that it can't be as good as it sounds. That's pretty much the reception cloud services got in the early days, circa 2010.

An indication of IT's growing acceptance of cloud services for mainstream applications is KPMG's annual survey of 650 enterprise executives in 16 countries about their cloud strategies. In the 2011 survey, concerns about data security, privacy, and regulatory compliance were cited as the principal impediments to cloud adoption in large organizations.

According to the results of the most recent KPMG cloud survey, executives now consider cloud integration challenges and control of implementation costs as their two greatest concerns. There's still plenty of fretting among executives about the security of their data in the cloud, however. Intellectual property theft, data loss/privacy, and system availability/business continuity are considered serious problems, according to the survey.

Executives rate such cloud-security challenges as intellectual property theft, data loss, and system availability greater than 4 on a scale of 1 (not serious) to 5 (very serious). Credit: KPMG

Still, security concerns aren't dissuading companies from adopting cloud services. Executives told KPMG that in the next 18 months their organizations planned cloud adoption in such areas as sourcing and procurement; supply chain and logistics; finance, accounting and financial management; business intelligence and analytics; and tax.

Cloud 'migration' is really a 'transformation'

Three business trends are converging to make the cloud an integral part of the modern organization: the need to collect, integrate, and analyze data from all internal operations; the need to develop applications and business processes quickly and inexpensively; and the need to control and monitor the use of data resources that are no longer stored in central repositories.

In a September 2, 2014, article on Forbes.com, Robert LeBlanc explains that cloud services were initially perceived as a way to make operations more efficient and less expensive. But now organizations see the cloud architecture as a way to innovate in all areas of the company. Business managers are turning to cloud services to integrate big data, mobile computing, and social media into their core processes.

Mobile and collaboration are leading the transition in organizations away from on-site management and toward cloud platforms. Credit: Ventana Research

George Washington University discovered first-hand the unforeseen benefits of its shift to a cloud-based data strategy. Zaid Shoorbajee describes in the March 3, 2014, GW Hatchet student newspaper how a series of campus-wide outages motivated the university to migrate some operations to cloud services. The switch saved the school $700,000 and allowed its IT staff to focus more on development and less on troubleshooting.

The benefits the school realized from the switch extend far beyond IT, however. Students now have the same "consumer and social experience" they've become accustomed to in their private lives through Google, iTunes, and similar services, according to a university spokesperson.

Four approaches to cloud application integration

Much of the speed, efficiency, and agility of cloud services can be lost when organizations become bogged down in their efforts to adapt legacy applications and processes. In a TechTarget article (registration required), Amy Reichert presents four approaches to cloud application integration. The process is anything but simple, due primarily to the nature of the applications themselves and the need to move data seamlessly and accurately between applications to support business processes.

One of the four techniques is labeled integration platform as a service (iPaas), in which the cloud service itself provides integration templates featuring such tools as connectors, APIs, and messaging systems. Organizations then customize and modify the templates to meet their specific needs.

In cloud-to-cloud integration, the organization's cloud applications have an integration layer built in to support any required data transformations, as well as encryption and transportation. The cloud-to-integrator-to-cloud model relies on the organization's existing middleware infrastructure to receive, convert, and transport the data between applications.

Finally, the hybrid integration approach keeps individual cloud apps separate but adds an integration component to each. This allows organizations to retain control over the data, maximize its investment in legacy systems, and adopt cloud services at the company's own pace.

Regardless of your organization's strategy for adopting and integrating cloud applications, the Morpheus database as a service can play a key role by providing a flexible, secure, and reliable platform for monitoring and optimizing database applications. Morpheus's SSD-backed infrastructure ensures lightning fast performance, and direct patches into EC2 offer ultra-low latency.

Morpheus protects your data via secure VPC connections and automatic backups, replication, and archiving. The service supports ElasticSearch, MongoDB, MySQL, and Redis, as well as custom storage engines. Create your free database during the beta period.

Daniel Radcliffe. Photograph: Perou/Guardian

TL;DR As the US State Department coped with massive database failure, thousands of travelers (and one Harry Potter star) were prevented entry to the United States. Even once the database was brought back online, it only worked in a limited capacity, resulting in extensive backlogs that added days, if not a full week, to wait time for visas and passports. Former U.S. Chief Technology Officer Todd Park wants government IT to move in the direction of open source, cloud-based computing. If you aren’t using SSD-backed cloud database infrastructure, it’s time to catch up.

The U.S. government might be able to afford the database downtime that most IT professionals price at $20,000 per hour minimum (some put that number in the hundreds of thousands), but chances are, most businesses are not equipped to suffer the consequences of database failure.

After the massive freak show that was the first iteration of Healthcare.gov, U.S. Chief Technology Officer (until a few days ago) Todd Park told Wired’s Steven Levy that he’d like to move government IT into the future with the rest of us, employing cloud-based, open source, rapid-iteration computing. He’s approached Silicon Valley tech gurus, imploring them to step in and champion for change. And he’s not shy about how dire the situation is: “America needs you! Not a year from now! But right. The. Fuck. Now!”

But that sort of modernization definitely hadn’t happened in Washington by mid-July, when State Department operations worldwide ground to a near standstill after the Consolidated Consular Database (CCD) crashed before their eyes.

As a result, an untold number of travelers were stuck waiting at embassies for authorizations that took, on average, a week longer to deliver than usual. Students, parents adopting new babies abroad, and vacationers alike found themselves trapped across the world from their destinations, all due to a system backup failure.

Database Crash Destroyed State Department Productivity

So what happened here? According to the DOS, an Oracle software upgrade “interfered with the smooth interoperability of redundant nodes.” On July 20, the Department began experiencing problems with the CCD, creating an ever-growing backlog of visa applications. While the applications were not lost, the crash rendered the CCD mostly unusable for at least a week.

Included in the backlog were only applications for non-immigrant visas. While the DOS would not confirm how many travelers were affected by the outage, State Department metrics show that 9 million non-immigrant visas were issued in 2012. Records are not yet available for more recent years, but United Press International reports that DOS issues 370,000 visas weekly and was operating at less than 50 percent productivity with a minimally functional system throughout the second half of July.

Nearly 9 million non-immigrant visas issued in 2012 alone. Backlogs due to database failure can be crippling. Credit: US Department of State

DOS’s Crashed Database Trapped Harry Potter, US Citizens, and Visitors Abroad

Daniel Radcliffe, forever known for his title role in the eight Harry Potter films, was among many people who faced impeded travel after the CCD failure. En route to Comic-Con in San Diego after a brief trip to Toronto for a premiere, even Radcliffe had to wait at the border due to delays in processing his new visa.

But while Radcliffe got an emergency pass, many less famous travelers weren’t so lucky. Several dozen families were living in a Guangzhou, China hotel after being unable to obtain visas for their newly adopted children. One Maryland family of seven was stuck for days, and they weren’t alone. The Washington Post reported that at least 30 other families were waiting, too, unable to return home as the DOS coped with the tech glitch.

Chinese students headed to the States for university studies were also delayed, alongside non-citizen Ireland residents traveling to the US for vacations. The US State Department’s Facebook page shows posts as late as August 22 asking for advice regarding delays in passport issuance.

Businesses Can’t Afford to Rely on Archaic Database Solutions

The Department of State posted an FAQ on August 4 in which they claimed that while they had made “significant progress,” they were still working to return the Consular Consolidated Database, brought back online July 23 only partially functional, back to “full operational capacity.” They still didn’t know the precise cause of the breakdown. The State Department hasn’t issued any statements since the August 4 update.

Needless to say, this debacle has caused a massive headache for the government and for travelers alike. But downtime causes headaches for companies in every industry. An International Data Corporation study reports that 98% of US and 96% of UK companies have to perform file recovery at least once per week. With downtime costing at least $20K per hour for both groups, and often considerably more, it’s imperative that businesses use database solutions that promise quick recovery. 

Downtime Costs in More Ways than One Credit: Emerson Network Power

Morpheus Backs Up Automatically and Is Lightning Fast

Clearly, few businesses can withstand the downtime from which the State Department continues to recover. You need your database to work quickly and reliably. Morpheus cloud database-as-a-service offers auto backups, replication, and archiving. Since it operates via online console, you’ll never have to worry about losing access to your systems and data. Its SSD-backed infrastructure increases IOPs by over 100 times, making it reliably fast. Direct connection to EC2 vastly reduces latency.

Todd Park wouldn’t want to move government IT to the cloud if he didn’t trust the security. Morpheus is secured by VPN and is safe from interference from the public internet. The Morpheus platform is continually monitored and managed by the sharply dedicated and experienced team at Morpheus, as well as sophisticated robots made for the job. You can also monitor in real time the queries that could potentially bog down your app performance. Support is available 24 hours a day.

Morpheus works with Elasticsearch, MongoDB, MySQL, and Redis. While Morpheus is in beta, you can try it at no cost. Prices after beta and supported platforms are listed on the Morpheus web site.

TL;DR: At one time, organizations planning their cloud strategy adopted an either-or approach: Either store and manage data on a secure private cloud, or opt for the database-as-a-service model of the public cloud. Now companies are realizing the benefits of both options by adopting a multicloud strategy that places individual applications on the platform that best suits them.

In IT's never-ending quest to improve database performance and reduce costs, a new tactic has surfaced: multicloud. Rather that process all database queries on either the private cloud or public cloud, shift the processing to the platform best able to handle it in terms of speed and efficiency.

InfoWorld's David Linthicum explains in an August 5, 2014, article that a multicloud architecture "gives those who manage large distributed databases the power to use only the providers who offer the best and most cost-effective service -- or the providers who are best suited to their database-processing needs."

Managing the resulting complexity isn't as daunting as it may sound, according to Linthicum. In fact, a cloud-management system could soon become a requirement for IT departments of all sizes. Product lifecycle management (PLM) expert Oleg Shilovitsky claims in an August 5, 2014, article on BeyondPLM.com that three trends are converging to make distributed database architectures mandatory.

The first trend is the tsunami of data that is overwhelming information systems and pushing traditional database architectures to their physical limits. The second trend is the increasingly distributed nature of organizations, which are adopting a design-anywhere, build-anywhere philosophy. The third trend is the demand among users for ever-faster performance on many different platforms to keep pace with the changes in the marketplace.

Multicloud: More than simply pairing public and private

In a July 12, 2013, article, InfoWorld's Linthicum compared the process of adopting a multicloud strategy to the transition a decade or more ago to distributed internal systems customized to the specific demands of the business. A key to managing the increased complexity of multicloud systems is carefully choosing your service provider to ensure a good fit between their offerings and your company's needs.

Three key considerations in this regard are security, accessibility, and scalability. These are three areas where the Morpheus database-as-a-service shines. In addition to lightning-fast SSD-based infrastructure that increases IOPs by 100 times, Morpheus provides real-time monitoring for identifying and optimizing database queries that are impeding database performance.

Morpheus offers ultra-low latency to leading Internet peering points and cloud hosts. Additionally, fault tolerance, disaster recovery, and automated backups make Morpheus a unique Database as a service. You connect to your databases via secure VPC. Visit the Morpheus site for pricing information or to create a free account during the beta period.

Mixing old and new while maximizing adaptability

Businesses of all types and sizes are emphasizing the ability to shift gears quickly in anticipation of industry trends. No longer can you simply react to market changes: You must be there ahead of the competition.

A principal benefit of the multicloud database architecture is flexibility. In an August 25, 2014, article on Forbes.com, IBM's Jeff Borek highlights the ability of multicloud databases to leverage existing IT infrastructure while realizing the agility, speed, and cost savings of cloud services.

A typical multicloud approach is use of the private cloud as a point-of-control interface to public cloud services. MSPMentor's Michael Brown describes such an architecture in an August 27, 2014, article.

Many companies use a private cloud to ensure regulatory compliance for storing health, financial, and other sensitive data. In such systems, the private cloud may serve as the gateway to the public cloud in a two-tier structure. In addition to providing a single interface for users, the two levels allow applications and processes to be customized for best fit while keeping sensitive data secure.

A multicloud-application prototype: Managing multiple application servers

There's no denying that managing a distributed database system is more complicated than maintaining the standard top-down RDBMS of yesteryear. In a July 23, 2013, article on GitHub, German Ramos Garcia presents a prototype multicloud application development model based on the Hydra service. The model addresses much of the complexity entailed in managing multiple application servers.

The web application is first divided into static elements (images, Javascript, static HTML, etc.), dynamic elements on a backend server, and a database to support the backend servers.

A prototype multicloud application architecture separates static, dynamic, and database-support servers.

The distributed architecture must provide mechanisms for controlling the various servers, balancing traffic between servers, and recovering from failures. It must also control sessions between servers and determine where to store application data.

An alternative approach to multicloud management is presented by Mauricio J. Rojas in a blog post from March 25, 2014. The model Rojas proposes is a mash-up of management tools from many different cloud services.

Management tools for distributed cloud-based databases should focus on user needs and offer best of breed from various providers.

Rojas recommends creating a single set of management components for both the public and private clouds. This allows you to "create the same conditions in both worlds" and move seamlessly between the public and private domains.

In addition to security, important considerations in developing a multicloud management system are auto-scaling and high availability. With the Morpheus database-as-a-service, you're covered in all three areas right out of the box--even Pinal Dave, the SQL Authority uses Morpheus.  Make Morpheus a key element of your multicloud strategy.

TL;DR: Following several high-profile development disasters, government IT departments have received a mandate to change their default app-development approach from the traditional top-down model to the agile, iterative, test-centric methodology favored by leading tech companies. While previous efforts to dynamite the entrenched, moribund IT-contracting process have crashed in flames, analysts hold out hope for the new 18F and U.S. Digital Service initiatives. Given the public's complete lack of faith in the government's ability to provide digital services, failure is simply not an option.

Can Silicon Valley save the federal government from itself? That's the goal of former U.S. Chief Technology Officer Todd Park, who relocated to California this summer and set about recruiting top-tier application developers from the most innovative tech companies on the planet to work for the government.

As Wired's Steven Levy reports in an August 28, 2014, article, Park hopes to appeal to developers' sense of patriotism. "America needs you," Levy quotes Park telling a group of engineers at the Mozilla Foundation headquarters. A quick review of recent federal-government IT debacles demonstrates the urgency of Park's appeal.

Start with the $300 million spent over the past six years by the Social Security Administration on a disability-claim filing system that remains unfinished. Then check out the FBI's failed Virtual Case File case-management initiative that had burnt through $600 million before being replaced by the equally troubled Sentinel system, as Jason Bloomberg explains in an August 22, 2012, CIO article.

But the poster child of dysfunctional government app development is HealthCare.gov., which Park was brought in to save after its spectacularly failed launch in October 2013. For their $300 million investment, U.S. taxpayers got a site that took eight seconds to respond to a mouse click and crashed so often that not one of the millions of people visiting the site on its first day of operation was able to complete an application.

Healthcare.gov's performance in the weeks after its launch highlight what can happen when a $300 million development project proceeds with no one in the driver's seat. Credit: The Verge

The dynamite approach to revamping government IT processes

Just months before HealthCare.gov's epic crash-and-burn, Park had established the Presidential Innovation Fellows program to attract tech professionals to six-month assignments with the government. The program was envisioned as a way to seed government agencies with people who could introduce cutting-edge tools and processes to their development efforts. After initial successes with such agencies as Medicare and Veterans Affairs, the group turned its attention to rescuing HealthCare.gov -- and perhaps the entire Affordable Care Act.

The source of the site's problems quickly became obvious: the many independent contractors assigned to portions of the site worked in silos, and no single contractor was responsible to ensure the whole shebang actually worked. Even as the glitches stacked up following the failed launch, contractors continued to work on new "features" because they were contractually required to meet specific goals.

The culprit was the federal contracting process. Bureaucrats farmed out contracts to cronies and insiders, whose only motivation was to be in good position to win the next contract put up for bid, according to Levy. Park's team of fixers was met with resistance at every turn despite being given carte blanche to ignore every rule of government development and procurement.

With persistence and at least one threat of physical force, the ad-hoc team applied a patchwork of monitoring, testing, and debugging tools that got the site operational. By April 2014, HealthCare.gov had achieved its initial goal of signing up 8 million people for medical insurance.

How an agile-development approach could save democracy

The silver lining of the HealthCare.gov debacle is the formation of two new departments charged with bringing an agile approach to government app development. The General Services Administration's 18F was established earlier this year with a mandate to "fail fast" rather than follow the standard government-IT propensity to fail big.

As Tech President's Alex Howard describes in an August 14, 2014, article, 18F is assisting agencies as they develop free, open-source services offered to the public via GitHub and other open-source repositories. Perhaps an even-bigger shift in attitude by government officials is the founding last month of the U.S. Digital Service, which is modeled after a successful U.K. government app-development program.

To help agencies jettison their old development habits in favor of modern approaches, the White House released the Digital Services Playbook that provides 13 "plays" drawn from successful best practices in the private and public sectors. Two of the plays recommend deploying in a flexible hosting environment and automating testing and deployment.

The government's Digital Services Playbook calls for agencies to implement modern development techniques such as flexible hosting and automated testing.

That's precisely where the Morpheus database-as-a-service (DBaas) fits into the government's plans. Morpheus lets users spin up a new database instance in seconds -- there's no need to wait for lengthy IT approval to procure and provision a new DB. Instead it's all done in the cloud within seconds.

In addition, users' core elastic, scalable, and reliable DB infrastructure is taken care for them. Developers can focus on building the core functionality of the app rather than having to spend their time making the infrastructure reliable and scalable. Morpheus delivers continuous availability, fault tolerance, fail over, and disaster recovery for all databases running on its service. Last but definitely not least, it's cost efficient for users to go with Morpheus: there's no upfront setup cost, and they pay only for actual usage.

The Morpheus cloud database as a service (DBaaS) epitomizes the goals of the government's new agile-development philosophy. The service's real-time monitoring makes continuous testing a fundamental component of database development and management. Morpheus's on-demand scalability ensures that applications have plenty of room to grow without incurring large up-front costs. You get all this plus industry-leading performance, VPN security, and automatic backups, archiving, and replication.

Government IT gets the green light to use cloud app-development services

As groundbreaking as the Digital Services Playbook promises to be for government IT, another publication released at the same time may have an even-greater positive impact on federal agencies. The TechFAR Handbook specifies how government contractors can support an "iterative, customer-driven software development process."

Tech President's Howard quotes Code for America founder Jen Pahlka stating that the handbook makes it clear to government IT staff and contractors alike that "agile development is not only perfectly legal, but [is] in fact the default methodology."

Critics point out that this is not the government's first attempt to make its application development processes more open and transparent. What's different this time is the sense of urgency surrounding efforts such as 18F and the U.S. Digital Service. Pahlka points out that people have lost faith in the government's ability to provide even basic digital services. Pahlka is quoted in a July 21, 2014, Government Technology interview by Colin Wood and Jessica Mulholland as stating, "If government is to regain the trust and faith of the public, we have to make services that work for users the norm, not the exception."

TL;DR: Securing your company's cloud-based assets starts by applying tried-and-true data-security practices modified to address the unique characteristics of virtual-network environments. Cloud services are slowly gaining the trust of IT managers who are justifiably hesitant to extend the security perimeters to accommodate placing their company's critical business assets in the cloud.

The fast pace of technological change doesn't faze IT pros, who live the axiom "The more things change, the more they stay the same." The solid security principles that have protected data centers for generations apply to securing your organization's assets that reside in the cloud. The key is to anticipate the new threats posed by cloud technology -- and by cyber criminals who now operate with a much higher level of sophistication.

In a September 18, 2014, article, ZDNet's Ram Lakshminarayanan breaks down the cloud-security challenge into four categories: 1) defending against cloud-based attacks by well-funded criminal organizations 2) unauthorized access and data breaches that use employees' stolen or compromised mobile devices 3) maintenance and monitoring of cloud-based APIs, and 4) ensuring compliance with the growing number and complexity of government regulations.

IT departments are noted for their deliberate approach to new technologies, and cloud-based data services are no different. According to a survey published this month by the Ponemon Institute of more than 1,000 European data-security practitioners (pdf), 64 percent believe their organization's use of cloud services reduces their ability to protect sensitive information.

The survey, which was sponsored by Netskope, blames much of the distrust on the cloud multiplier effect: IT is challenged to track the increasing number and type of devices connecting to the company's networks, as well as the cloud-hosted software employees are using, and the business-critical applications being used in the "cloud workspace."

Building trust between cloud service providers and their IT customers

No IT department will trust the organization's sensitive data to a service that fails to comply with privacy and data-security regulations. The Ponemon survey indicates that cloud services haven't convinced their potential customers in Europe of their trustworthiness: 72 percent of respondents strongly disagreed, disagreed, or were uncertain whether their cloud-service providers were in full compliance with privacy and data-security laws.

Data-security executives remain leery of cloud services' ability to secure their organization's critical business data. Credit: Ponemon Institute

Even more troubling for cloud service providers is the survey finding that 85 percent of respondents strongly disagreed, disagreed, or weren't sure whether their cloud service would notify them immediately in the event of a data breach that affected their company's confidential information or intellectual property.

The Morpheus database-as-a-service puts data security front and center by offering VPN connections to your databases in addition to online monitoring and support. Your databases are automatically backed up, replicated, and archived on the service's SSD-backed infrastructure.

Morpheus also features market-leading performance, availability, and reliability via direct connections to EC2 and colocation with the fastest peering points available. The service's real-time monitoring lets you identify and optimize the queries that are slowing your database's performance. Visit the Morpheus site for pricing information and to sign up for a free account.

Overcoming concerns about cloud-service security

Watching your data "leave the nest" can be difficult for any IT manager. Yet cloud service providers offer a level of security at least on par with that of their on-premises networks. In a September 15, 2014, article on Automated Trader, Bryson Hopkins points out that Amazon Web Services and Microsoft Azure are two of the many public cloud services that comply with Service Organization Control (SOC), HIPPA, FedRAMP, ISO 27001, and other security standards.

The SANS Institute's Introduction to Securing a Cloud Environment (pdf) explains that despite the cloud's increased "attack surface" when compared with in-house servers, the risk of cloud-based data being breached is actually less than that of losing locally hosted data. Physical and premises security are handled by the cloud service but can be enhanced by applying a layered approach to security that uses virtual firewalls, security gateways, and other techniques.

Cloud services avoid resource contention and other potential problems resulting from multi-tenancy by reprovisioning virtual machines, overprovisioning to crowd out other tenants, and using fully reserved capacities.

Another technique for protecting sensitive data in multi-tenant environments is to isolate networks by configuring virtual switches or virtual LANs. The virtual machine and management traffic must be isolated from each other at the data link layer (layer 2) of the OSI model.

The key to protecting sensitive data in a multi-tenant cloud environment is to isolate virtual machine and management traffic at the data link layer. Credit: SANS Institute

In a June 27, 2014, article on CloudPro, Davey Winder brings the issue of cloud security full circle by highlighting the fact that the core principles are the same as for other forms of data security: an iron-clad policy teamed with encryption. The policy must limit privileged-user access by the service's employees and provide a way for customers to audit the cloud network.

One way to compare in-house data management and cloud-based management is via the farmer-restaurant analogy described in a September 15, 2014, article by Arun Anandasivam on IBM's Thoughts on Cloud site. If you buy your food directly from the farmer, you have a first-hand impression of the person who grew your food, but your options may be limited and you have to do the preparation work. If you buy your food from a restaurant, you likely have a wider selection to choose from and you needn't prepare the meal, but you have less control over the food's path from farm to kitchen, and you have fewer opportunities to determine beforehand whether the food meets your quality requirements.

That's not to say farmers are any more or less trustworthy than restaurants. You use the same senses to ensure you're getting what you paid for, just in different ways. So check out the Morpheus database-as-a-service to see what's on the menu!

TL;DR: Old divisions in IT departments between app development and operations are crashing to the ground as users demand more apps with more features, and right now! By combining agile-development techniques and a hybrid public-private cloud methodology, companies realize the benefits of new technologies and place IT at the center of their operations.

The re-invention of the IT department is well underway. The end result will put technology at the core of every organization.

Gone are the days when IT was perceived as a cost center whose role was to support the company's revenue-generating operations. Today, software is imbued in every facet of the organization, whether the company makes lug nuts or space crafts, lima beans or Linux distros.

The nexus of the IT transformation is the intersection of three disparate-yet-related trends: the merger of development and operations (DevOps), the wide-scale adoption of agile-development methodologies, and the rise of hybrid public/private clouds.

In a September 12, 2014, article, eWeek's Chris Preimesberger quotes a 2013 study by Puppet Labs indicating the switch to DevOps is well underway: 66 percent of the organizations surveyed had adopted DevOps or planned to do so, and 88 percent of telecoms use or intend to use a DevOps approach. The survey also found that DevOps companies deploy code 30 times more frequently than their traditional counterparts.

Closing the loop that links development and operations

A successful DevOps approach requires a closed loop connecting development and operations via continuous integration and continuous deployment. This entails adoption of an entirely new and fully automated development toolset. Traditional IT systems simply can't support the performance, scalability, and latency requirements of a continuous-deployment mentality. These are the precise areas where cloud architectures shine.

Agile development combines with DevOps to create a service-based approach to the provisioning, support, and maintenance of apps. Source: Dev2Ops

For example, the Morpheus database-as-a-service offers ultra-low latency via direct patches into EC2 and colocation with among the fastest peering points available. You can monitor and optimize your apps in real time and spot trends via custom metrics. Morpheus's support staff and advanced robots monitor your database infrastructure continuously, and custom MongoDB and MySQL storage engines are available.

In addition, you're assured high availability via secure VPC connections to the network, which uses 100-percent bare-metal SSD storage. Visit the Morpheus site for pricing information and to sign up for a free account.

Continuous integration + continuous delivery = continuous testing

Developers steeped in the tradition of delivering complete, finished products have to turn their thinking around 180 degrees. Dr. Dobb's Andrew Binstock explains in a September 16, 2014, article that continuous delivery requires deploying tested, usable apps that are not feature-complete. The proliferation of mobile and web interfaces makes constant tweaks and updates not only possible but preferable.

Pushing out 10 or more updates in a day would have been unheard of in a turn-of-the-millennium IT department. The incessant test-deploy-feedback loop is possible only if developers and operations staff work together to ensure smooth roll-outs and fast, effective responses to the inevitable deployment errors and other problems.

Integrating development and operations so completely requires not just a reorganization of personnel but also a change in management philosophy. However, the benefits of such a holistic approach to IT outweigh the short-term pain of the organizational adjustments required.

A key to smoothing out some of the bumps is use of a hybrid-cloud philosophy that delivers the speed, scalability, and cost advantages of the public cloud while shielding the company's mission-critical applications from the vagaries of third-party platforms. Processor, storage, and network resources can be provisioned quickly as services by using web interfaces and APIs.

Seeing apps as a collection of discrete services

Imagine a car that's still drivable with only three of its four wheels in place. That's the idea behind developing applications as a set of discrete services, each of which is able to function independently of the others. Also, the services can be swapped in and out of apps on demand.

This is the "microservice architecture" described by Martin Fowler and James Lewis in a March 25, 2014, blog post. The many services that comprise such an app run in their own processes and communicate via an HTTP resource API or other lightweight mechanism. The services can be written in different programming languages and can use various storage technologies because they require very little centralized management.

The microservice architecture separates each function of the app as a separate service rather than encapsulating all functions in a single process. Source: Martin Fowler

By using services rather than libraries as components, the services can be deployed independently. When a service changes, only that service needs to be redeployed -- with some noteworthy exceptions, such as changes to service interfaces.

No longer are applications "delivered" by developers to users. In the world of DevOps, the team "developing" the app owns it throughout its lifecycle. Thus the "developers" take on the sys-admin and operations support/maintenance roles. Gone are the days of IT working on "projects." Today, all IT staff are working on "products." This cements to position the company's technology workers at the center of all the organization's operations.

TL;DR: Google Analytics stores a massive amount of statistical data from web sites across the globe. Retrieving reports quickly from such a large amount of data requires Google to use a custom solution that is easily scalable whenever more data needs to be stored.

At Google, any number of applications may need to be added to their infrastructure at any time, and each of these could potentially have extremely heavy workloads. Resource demands such as these can be difficult to meet, especially when there is a limited amount of time to get the required updates implemented.

If Google were to use the typical relational database on a single server node, they would need to upgrade their hardware each time capacity is reached. Given the amount of applications being created and data being used by Google, this type of upgrade could quite possibly be necessary on a daily basis!

The load could also be shared across multiple server nodes, but once more than a few additional nodes are required, the complexity of the system becomes extremely difficult to maintain.

With these things in mind, a standard relational database setup would not be a particularly attractive option due to the difficulty of upgrading and maintaining the system on such a large scale.

Finding a Scalable Solution

In order to maintain speed and ensure that such incredibly quick hardware upgrades are not necessary, Google uses its own data storage solution called BigTable. Rather than store data relationally in tables, it stores data as a multi-dimensional sorted map.

This type of implementation falls under a broader heading for data storage, called a key/value store. This method of storage can provide some performance benefits and make the process of scaling much easier.

Information Storage in a Relational Database

Relational databases store each piece of information in a single location, which is typically a column within a table. For a relational database, it is important to normalize the data. This process ensures that there is no duplication of data in other tables or columns.

For example, customer last names should always be stored in a particular column in a particular table. If a customer last name is found in another column or table within the database, then it should be removed and the original column and table should be referenced to retrieve the information.

The downside to this structure is that the database can become quite complex internally. Even a relatively simple query can have a large number of possible paths for execution, and all of these paths must be evaluated at run time to find out which one will be the most optimal. The more complex the database becomes, the more resources will need to be devoted to determining query paths at run time.

Information Storage in a Key/Value Store

With a key/value store, duplicate data is acceptable. The idea is to make use of disk space, which can easily and cost-effectively be upgraded (especially when using a cloud), rather than other hardware resources that are more expensive to bring up to speed.

This data duplication is beneficial when it comes to simplifying queries, since related information can be stored together to avoid having numerous potential paths that a query could take to access the needed data.

Instead of using tables like a relational database, key/value stores use domains. A domain is a storage area where data can be placed, but does not require a predefined schema. Pieces of data within a domain are defined by keys, and these keys can have any number of attributes attached to them.

The attributes can simply be string values, but can also be something even more powerful: data types that match up with those of popular programming languages. These could include arrays, objects, integers, floats, Booleans, and other essential data types used in programming.

With key/value stores, the data integrity and logic are handled by the application code (through the use of one or more APIs) rather than by using a scheme within the database itself. As a result, data retrieval becomes a matter of using the correct programming logic rather than relying on the database optimizer to determine the query path from a large number of possibilities based on the relation it needs to access.

Getting Results

Google needs to store and retrieve copious amounts of data for many applications, included among them are Google Analytics, Google Maps, Gmail, and their popular web index for searching. In addition, more applications and data stores could be added at any time, making their BigTable key/value store an ideal solution for scalability.

BigTable is Google’s own custom solution, so how can a business obtain a similar performance and scalability boost to give its users a better experience? The good news is that there are other key/value store options available, and some can be run as a service from a cloud. This type of service is easily scalable, since more data storage can easily be purchased as needed on the cloud.

A Key/Value Store Option

There are several options for key/value stores. One of these is Mongo, which is designed as an object database that stores information in JSON format. This format is ideal for web applications since JSON data makes it easy to pass data around in a standard format among the various parts of an application that need it.

For example, Mongo is part of the MEAN stack: Mongo, Express, AngularJS, and NodeJS—a popular setup for programmers developing applications. Each of these pieces of the puzzle will send data to and from other one or more of the other pieces. Since everything, including the database, can use the JSON format, passing the data around among the various parts becomes much easier and more standardized.

How to Make Use of Mongo

Mongo can be installed and used on various operating systems, including Windows, Linux, and OS X. In this case, the scalability of the database would need to be maintained by adding storage space to the server on which it is installed.

Another option is to use Mongo as a service on the cloud. This allows for easy scalability, since a request can be made to the service provider to up the necessary storage space at any time. In this way, new applications or additional data storage needs can be handled quickly and efficiently.

Morpheus is a great option for this service. Mongo is offered, as well as a number of other databases. Using Morpheus, a highly scalable database as a service can be running in no time!