• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
  • Articles
  • News
  • Events
  • Advertize
  • Jobs
  • Courses
  • Contact
  • (0)
  • LoginRegister
    • Facebook
    • LinkedIn
    • RSS
      Articles
      News
      Events
      Job Posts
    • Twitter
Datafloq

Datafloq

Data and Technology Insights

  • Categories
    • Big Data
    • Blockchain
    • Cloud
    • Internet Of Things
    • Metaverse
    • Robotics
    • Cybersecurity
    • Startups
    • Strategy
    • Technical
  • Big Data
  • Blockchain
  • Cloud
  • Metaverse
  • Internet Of Things
  • Robotics
  • Cybersecurity
  • Startups
  • Strategy
  • Technical

Big Data and Risk Management in Financial Markets (Part II)

Francesco Corea / 8 min read.
April 23, 2017
Datafloq AI Score
×

Datafloq AI Score: 85.33

Datafloq enables anyone to contribute articles, but we value high-quality content. This means that we do not accept SEO link building content, spammy articles, clickbait, articles written by bots and especially not misinformation. Therefore, we have developed an AI, built using multiple built open-source and proprietary tools to instantly define whether an article is written by a human or a bot and determine the level of bias, objectivity, whether it is fact-based or not, sentiment and overall quality.

Articles published on Datafloq need to have a minimum AI score of 60% and we provide this graph to give more detailed information on how we rate this article. Please note that this is a work in progress and if you have any suggestions, feel free to contact us.

floq.to/he4Le

I. Introduction to forecasting

If you missed the first part, I suggest you read it before going through this article. It gives a good introduction as well as an overview of traditional risk management and big data simulation. This article is instead more focused on big data forecasting.

There are nowadays several new techniques or methods borrowed from other disciplines which are used by financial practitioners with the aim of predicting future market outcomes.

Eklund and Kapetanios (2008) provided a good review of all the new predictive methods and I am borrowing here their classification of forecasting methods, which divides techniques into four groups: single equation models that use the whole datasets; models that use only a subset of the whole database, even though a complete set is provided; models that use partial datasets to estimate multiple forecasts averaged later on; and finally, multivariate models that use the whole datasets.


II. Single equation models

The first group is quite wide and includes common techniques used differently, such as ordinary least square (OLS) regression or Bayesian regression, as well as new advancements in the field, as in the case of factor models.

In the OLS model, when the time series dimension exceeds the number of observations, the generalized inverse has to be used in order to estimate the parameters.

Bayesian regression (De Mol, Giannone, and Reichlin, 2007) starts instead from a prior probability and updates this likelihood through an incremental amount of observations to finally obtain a posterior probability. Of particular interest in this type of regression is the ability to use the whole dataset reducing though the magnitude of parameters, and in this way providing a lower variance with respect to standard estimators.

Finally, a part of this first group is the factor models (Stock and Watson, 2002). These models are able to select ex-ante which data contains the greatest predictive power and use then only a few variables to form the forecasting equation. Famous models of this kind are the principal component analysis (PCA), its dynamic version, and the subspace methods.

The PCA technique estimates the matrix of the linear combinations of all factors through their eigenvectors and then considers only the first k ones with the greatest weight (loading vector). If in place of the covariance matrix the spectral density one with different frequencies is used, the technique is called dynamic PCA.

The subspace method starts instead from a parametric state-space model: a simple multivariate OLS model estimates the coefficient, while the factors are then obtained through a reduced rank approximation (Kapetanios and Marcellino, 2003).


III. Subset models

In the second group, we expect the most of the data to be noisy and not really meaningful, thus we use features or variables selection models to identify ex-ante the more significant predictors. This class of methods provides then a stratagem to avoid (if wanted) the problem to deal with large datasets.

The dimensionality reduction allows indeed the use of the simple linear forecasting model on the most appropriate subset of variables, which are skimmed down through information criteria. Boosting, LASSO, Least Angle Regression (LAR), are some of the most common techniques belonging to this class.

Boosting entails running univariate regressions for each predictor, and selecting the model that minimizes a chosen loss function. The residuals so obtained are then explained through running the second round of regressions using the remaining predictors, and selecting again the one that minimizes the same loss function as before. The process is repeated until a certain information criterion is met.

On the other hand, the LASSO and LAR applied penalty functions to the regressions. LASSO expects the norm of estimated vector to be less than a specified shrinkage threshold; the LAR works similarly to the boosting, even though at each step is not included a new variable, but the relative coefficient is incremented by the amount that makes it not to minimize the loss function anymore.

Many other procedures exist, such as stepwise regression, ridge regression, or still the more exotic genetic algorithm and simulated annealing (Kapetanios, 2007).

The stepwise regression estimates the model starting from no variable (forward regression) or from an all-variable model (backward), adding and subtracting at each step the variable that improves the model the most and keeping repeating the procedure until the process stops.

The ridge regression instead includes a quadratic term, which penalizes the size of the regression coefficients.

The genetic algorithm (Dorsey and Mayer, 1995) is an optimization method that iterates towards a solution selecting, in the same fashion of natural selection, only the best fitting features.

Finally, simulated annealing creates a non-homogeneous Markov chain using the objective function that converges to a maximum/minimum of the function itself.


IV. Partial datasets for multiple forecasts

The third group is the averaging model one, which embeds mainly the Bayesian model averaging (BMA) and frequentist model averaging.


Interested in what the future will bring? Download our 2023 Technology Trends eBook for free.

Consent

While the frequentist approach entails the creation of model confidence sets from which model likelihood can be derived, the BMA methodology provides a Bayesian framework for the combination models.

Different combinations of relationships between predictors and dependent variables are estimated and weighted altogether in order to obtain a better forecasting model, with weights corresponding to the posterior probabilities.


V. Multivariate models

Finally, the last group uses the whole datasets to estimate a set of variables (Carriero, Kapetanios, and Marcellino, 2007). Reduced rank regression, Bayesian VAR, and Multivariate Boosting, as already proposed in Eklund and Kapetanios (2008), are only a few of the models belonging to this class.

Reduced rank regression works similarly to a set of classic Vector Autoregressive models, but as soon as the underlying dataset is large, those models become quite noisy and rich of insignificant coefficients. Therefore, a rank reduction can be imposed, constraining the matrix of the coefficients in the VAR model to be much smaller than the number of predictors.

If the previous model compressed the greatest informative value into few predictors, the Bayesian counterpart of the VAR focuses instead on constraining the data imposing the restrictions as priors although it maintains a dependency between data and coefficient determination.

Multivariate boosting is quite similar to its simple version explained beforehand. The main difference lies in measuring at each step a multivariate model (instead of a single equation as in simple boosting), starting from a zero coefficient matrix and recursively setting the single coefficients that explained better the dependent variables to be non-zero.


VI. Other models

Many models have been provided in the previous sections, but the list is not exhaustive. Indeed, according to Varian (2014), classification and regression trees (CART), bagging, bootstrapping, random forests and spike and slab regression are other models commonly used in machine learning applications for finance.

Decision trees are used when it is necessary to describe a sequence of decisions/outcomes in a discrete way, and they work quite well in nonlinear cases and with missing data issue.

Bootstrapping estimates the sampling distribution of some statistics through repeated sampling with replacement, while bagging averages different models obtained with manifold bootstrapping of different size.

The random forests instead is a machine learning technique that grows a tree starting from a bootstrap sample and selects at each node a random sample of predictors. This process is repeated several times, and a majority vote is applied for classification purposes.

Finally, spike and slab regression is another Bayesian hierarchical model that specifies a (Bernoulli) prior distribution of the probability of a set of variables to be included in the model this is called spike, i.e., the probability of a coefficient to be non-zero. Afterwards, a second prior is selected, i.e. a prior on the regression coefficient deriving from the previous choice of certain variables (the slab). Combining the priors and repeating the process multiple times, it is possible to obtain two posterior distributions for both the priors, as well as the prediction of the overall model.


VII. Final considerations

Varian (2014) suggests a few additional points that should be considered as the dataset becomes larger: causal inference (Angrist and Pischke, 2009) and model uncertainty.

Causation and correlation are two far distinct concepts, and the issue with large noisy datasets is that is much easier to spot out spurious correlations that do not have real meaning or practical validation. Random-control groups may be necessary in those cases, but sometimes a good predictive model can even work better (Varian, 2014).

Causation and correlation are two distinct concepts

The second feature pointed out is that, when it comes to large datasets, averaging models may be more effective than choosing a single one. Usually a simple unique representation is analyzed because of data scarcity, but nothing prevents the decision maker from using multiple models as soon as more data allow it.

References

  1. Angrist, J. D., Pischke, J. S. (2009). Mostly Harmless Econometrics . Princeton University Press.
  2. Carriero, A., Kapetanios, G., Marcellino, M. (2011). Forecasting Large Datasets with Bayesian Reduced Rank Multivariate Models . Journal of Applied Econometrics 26 (5): 735 ‘761.
  3. De Mol, C., D. Giannone, and L. Reichlin (2008): Forecasting using a large number of predictors: is Bayesian regression a valid alternative to principal components? . Journal of Econometrics 146: 318 ‘328.
  4. Dorsey, R. E., Mayer, W. J. (1995). Genetic Algorithms for Estimation Problems with Multiple Optima, Nondifferentiability and Other Irregular Features . Journal of Business and Economic Statistics, 13 (1): 53 ’66.
  5. Eklund, J., Kapetanios, G. (2008). A Review of Forecasting Techniques for Large Data Sets . Queen Mary working paper no. 625: 1 ’18.
  6. Kapetanios, G. (2007). Variable Selection in Regression Models using Non- Standard Optimisation of Information Criteria . Computational Statistics & Data Analysis 52: 4 ’15.
  7. Kapetanios, G., and M. Marcellino. (2003). A Comparison of Estimation Methods for Dynamic Factor Models of Large Dimensions. Queen Mary, University of London Working Paper 489.
  8. Stock, J. H., Watson, M. W. (2002). Macroeconomic Forecasting Using Diffusion Indices. Journal of Business and Economic Statistics, 20, 147 ‘162.
  9. Varian, H. R. (2014). Big Data: New Tricks for Econometrics. Journal of Economic Perspectives, 28 (2): 3 ’28.

Note: I have written an extended technical survey on big data and risk management for the Montreal Institute of Structured Finance and Derivatives (2016). If you want to read the complete work, here it comes the pdf.

‘ ‘

Follow me on Medium

Categories: Big Data, Technical
Tags: Artificial Intelligence, Big Data, financial, models, risks

About Francesco Corea

Editor at Cyber Tales. Complexity scientist and data strategist, Francesco is a strong supporter of an interdisciplinary research approach, and he wants to foster the interaction of different sciences in order to bring to light hidden connections. He is a former Anthemis Fellow, IPAM Fellow, and a PhD graduate at LUISS University. His topics of interests are big data and AI, and he focuses on fintech, medtech, and energy verticals.

Primary Sidebar

E-mail Newsletter

Sign up to receive email updates daily and to hear what's going on with us!

Publish
AN Article
Submit
a press release
List
AN Event
Create
A Job Post
Host your website with Managed WordPress for $1.00/mo with GoDaddy!

Related Articles

The Advantages of IT Staff Augmentation Over Traditional Hiring

May 4, 2023 By Mukesh Ram

The State of Digital Asset Management in 2023

May 3, 2023 By pimcoremkt

Test Data Management – Implementation Challenges and Tools Available

May 1, 2023 By yash.mehta262

Related Jobs

  • Software Engineer | South Yorkshire, GB - February 07, 2023
  • Software Engineer with C# .net Investment House | London, GB - February 07, 2023
  • Senior Java Developer | London, GB - February 07, 2023
  • Software Engineer – Growing Digital Media Company | London, GB - February 07, 2023
  • LBG Returners – Senior Data Analyst | Chester Moor, GB - February 07, 2023
More Jobs

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto customers Data design development digital environment experience future Google+ government information learning machine learning market mobile Musk news Other public research sales security share social social media software strategy technology twitter

Related Events

  • 6th Middle East Banking AI & Analytics Summit 2023 | Riyadh, Saudi Arabia - May 10, 2023
  • Data Science Salon NYC: AI & Machine Learning in Finance & Technology | The Theater Center - December 7, 2022
  • Big Data LDN 2023 | Olympia London - September 20, 2023
More events

Related Online Courses

  • Oracle Cloud Data Management Foundations Workshop
  • Data Science at Scale
  • Statistics with Python
More courses

Footer


Datafloq is the one-stop source for big data, blockchain and artificial intelligence. We offer information, insights and opportunities to drive innovation with emerging technologies.

  • Facebook
  • LinkedIn
  • RSS
  • Twitter

Recent

  • 5 Reasons Why Modern Data Integration Gives You a Competitive Advantage
  • 5 Most Common Database Structures for Small Businesses
  • 6 Ways to Reduce IT Costs Through Observability
  • How is Big Data Analytics Used in Business? These 5 Use Cases Share Valuable Insights
  • How Realistic Are Self-Driving Cars?

Search

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto customers Data design development digital environment experience future Google+ government information learning machine learning market mobile Musk news Other public research sales security share social social media software strategy technology twitter

Copyright © 2023 Datafloq
HTML Sitemap| Privacy| Terms| Cookies

  • Facebook
  • Twitter
  • LinkedIn
  • WhatsApp

In order to optimize the website and to continuously improve Datafloq, we use cookies. For more information click here.

Dear visitor,
Thank you for visiting Datafloq. If you find our content interesting, please subscribe to our weekly newsletter:

Did you know that you can publish job posts for free on Datafloq? You can start immediately and find the best candidates for free! Click here to get started.

Not Now Subscribe

Thanks for visiting Datafloq
If you enjoyed our content on emerging technologies, why not subscribe to our weekly newsletter to receive the latest news straight into your mailbox?

Subscribe

No thanks

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

Marketing cookies

This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Keeping this cookie enabled helps us to improve our website.

Please enable Strictly Necessary Cookies first so that we can save your preferences!