Tag Archives: Agile

Designing the future – Data Innovation Labs

With the ongoing Big data revolution, and the impending Internet of Things revolution, there has been a renewed enthusiasm in “innovation” around data.  Similar to the Labs concept started by Google (think Gmail Beta based on Ajax, circa 2004), more and more organizations, business communities, governments and countries are setting up Labs to foster innovation in data and analytics technologies. The idea behind these “data innovation labs” is to develop avant-garde data and analytics technologies and products in an agile fashion and move quickly from concept to production. Given the traditional bureaucratic setup in large organizations and governments, these Labs stand a better chance of fostering a culture of innovation, due to their being autonomous entities and their startup-mode culture leveraging agile methodologies.

Data Innovation Labs

Data Innovation Labs

Below I list a few of the data innovation labs that have been setup to get value for their parent entities in the data and analytics space, trying to build data products in the Big Data and Data Science fields.

  • Data Innovation Lab – Thomson Reuters
    • Small group of about ten people, partnering with internal teams, third-parties and customers to find data-driven innovations
    • experiments with mash-ups of internal and external data in novel ways
    • hosts internal crowd-sourcing competitions
    • translate business problem into technical data problem statement
    • created Exchange – a digital forum for sharing ideas and insights
    • Partners with Central Strategy to estimate potential and market-size for new data innovation opportunities
  • The Data Lab – Scotland
    • mission is to strengthen Scotland’s local industry and transfer world-leading research in informatics and computer science in the global marketplace
    • focuses on skills and training by working with industry  to create  a pipeline of talented data scientists equipped with the relevant skills
    • connects world-leading researchers and data scientists with local industry and public sector organizations, giving them access to experts who can help collaborate on solutions to key problems
  • Smart Data Innovation Lab – German government
    • Hosted at Karlsruhe Institute for Technology, its mission is to turn big data into smart data
    • plans to store data centrally in a highly secured environment for research purposes
    • has cutting-edge insfrastructure for processing Big Data including software like SAP HANA, IBM Watson and hardware on IBM Power and Intel architectures
    • Industry partners to deliver data sources directly from the practice environment, to be complemented with crowdsourced data and open data
    • plans to offer an open source repository for reuse in research
  • Midata Innovation Lab – UK government
    • An organization run by the Department of  Business Innovation & Skills with involvement of industry
    • Accelerator for businesses to create new services for consumers, from data
    • work involves concept of personal data stores (PDS) or personal clouds
    • working with three leading PDS – Allfiled, Mydex and Paoga
    • Participating organizations and developers use PDS to create new innovative services for consumers
  • Nordstrom Innovation Lab – Nordstrom
    • Internal technology lab focused on innvoation around technology
    • Secondary focus – but still in scope: operations, products, business models and management
    • Goal is to deliver data-driven products to inform business decisions internally, and to enhance customer experience externally
    • Multi-disciplinary team of techies, designers, entrepreneurs, statisticians, researchers and artists
  • GFDRR Innovation Lab – World Bank
    • Global facility for disaster reduction and recovery, a global partnership, managed by the World Bank and funded by 25 donor partners
    • supports use of science, technology, open data and innovation to empower decision-makers to increase their resilience
    • tries to apply the concepts of the global open data movement to the challenges of reducing vulnerability to natural hazards and  the impacts of climate change through OpenDRI (Open Data for Resilience Initiative)
  • Big Data Innovation Center and Innovation Lab – SAP
    • Focus on SAP’s mobile and cloud portfolio
    • Mission is to extend SAP stack and develop innovative data-driven process applications leveraging an integrated platform and next-generation DB technologies
    • Partnership and exchanges with leading schools including Stanford, MIT, Berlin universities
    • Short, fast-paced innovation cycles
    • Project run-times of a few months on an average
    • Hands over prototypes to SAP development for turning into market-ready products

Agile Development for BI

How can you reduce development costs and improve software reliability and accuracy at the same time? How can you make IT work together with Business while architect-ing your BI applications? If these goals sound contradictory and difficult to achieve, then Agile development may well fit the bill. Indeed in numerous BI projects, one or the other flavor of Agile is used to attain these very goals.

Defining Agile
There are several Agile development methodologies available:

• eXtreme Programming (XP)
• Scrum
• Kanban
• Feature-Driven Development (FDD)
• Crystal Clear
• Dynamic Systems Development Method (DSDM)
• Adaptive Software Development (ASD) and more…

At the core of any flavor of Agile development methodology is the iteration, which may last from 1 to 4 weeks (one unit of time) to develop a piece of the software. Each iteration is treated as an entire software project with its associated planning, design, coding, testing and documentation tasks.

What is it about Agile development which makes it particularly suitable for data warehousing and business intelligence projects?

* Agile emphasizes on communication be it through meetings (be it through the phone, VOIP, web or IM) over written documents. The idea is to get the user involved much early in the development process and incorporate their feedback, so as to minimize the risk of developing faulty software. For organizations adopting BI, very often users are clueless about the systems to build, the technology to use or even the range of analysis they require. Products are often bought after effective sales pitches from vendors and left to IT to deploy and architect. In such cases, IT can use Agile methodologies like DSDM, SCRUM or ASD to flesh out the requirements and deliver BI which actually provides insight rather than building a monolithic and unreliable data warehouse difficult to query and administer.

* Agile gels well with the evolutionary approach required for a data warehousing / BI lifecycle. Requirements change over time, and the iterations of the Agile methodology (with database re-factoring and evolutionary data modeling ) is more efficient in capturing these changes than the classical waterfall approach.

* Proof of the concept, technology and architecture is crucial to justify continued investment in DW/BI projects, especially on the enterprise scale. This is simpler and easier to do with Agile.

* Agile imbibes every member of the project team with extra responsibilities, making them owners of discrete functions and helps the project manager overcome the ‘taskmaster‘ stereotype and concentrate on being a leader or a visionary.

BI is essentially gaining competitive edge by insight into your business through lagging (measures) and leading (predictive model-based) metrics, which allows feedback cycles and restructuring of processes (Plan-Do-Check-Act Deming cycle). This essentially involves cooperation and teamwork across functions to model and understand the multi-faceted perspectives. Teamwork being the foundation of Agile, it is a natural fit for projects in BI and data warehousing.