bibliography

  1. ⊞ forecast: Forecasting functions for time series and linear models (2018)
    [1]
    R. Hyndman et al., forecast: Forecasting functions for time series and linear models. 2018.
    00000
    link
    BIB
    @book{hyndman2018forecast:, title = {forecast: {Forecasting} functions for time series and linear models}, url = {http://pkg.robjhyndman.com/forecast}, author = {Hyndman, Rob and Bergmeir, Christoph and Caceres, Gabriel and Chhay, Leanne and O'Hara-Wild, Mitchell and Petropoulos, Fotios and Razbash, Slava and Wang, Earo and Yasmeen, Farah}, year = {2018}, note = {00000} }
    RIS
    IEEE
    [1]
    R. Hyndman et al., forecast: Forecasting functions for time series and linear models. 2018.
  2. ⊞ Population Based Training of Neural Networks arXiv:1711.09846 [cs] (November 2017)
    [1]
    M. Jaderberg et al., “Population Based Training of Neural Networks,” arXiv:1711.09846 [cs], Nov. 2017.
    arXiv:1711.09846 [cs]
    link
    BIB
    @article{jaderberg2017population, title = {Population {Based} {Training} of {Neural} {Networks}}, url = {http://arxiv.org/abs/1711.09846}, journal = {arXiv:1711.09846 [cs]}, author = {Jaderberg, Max and Dalibard, Valentin and Osindero, Simon and Czarnecki, Wojciech M. and Donahue, Jeff and Razavi, Ali and Vinyals, Oriol and Green, Tim and Dunning, Iain and Simonyan, Karen and Fernando, Chrisantha and Kavukcuoglu, Koray}, month = nov, year = {2017} }
    RIS
    IEEE
    [1]
    M. Jaderberg et al., “Population Based Training of Neural Networks,” arXiv:1711.09846 [cs], Nov. 2017.
  3. ⊞ Macroeconomic Indicator Forecasting with Deep Neural Networks (September 2017)
    [1]
    A. Smalter Hall and T. R. Cook, “Macroeconomic Indicator Forecasting with Deep Neural Networks,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3046657, Sep. 2017.
    Abstract
    Economic policymaking relies upon accurate forecasts of economic conditions. Current methods for unconditional forecasting are dominated by inherently linear models that exhibit model dependence and have high data demands. We explore deep neural networks as an opportunity to improve upon forecast accuracy with limited data and while remaining agnostic as to functional form. We focus on predicting civilian unemployment using models based on four different neural network architectures. Each of these models outperforms benchmark models at short time horizons. One model‚ based on an Encoder Decoder architecture outperforms benchmark models at every forecast horizon (up to four quarters).
    link
    BIB
    @techreport{smalter_hall_macroeconomic_2017, address = {Rochester, NY}, type = {{SSRN} {Scholarly} {Paper}}, title = {Macroeconomic {Indicator} {Forecasting} with {Deep} {Neural} {Networks}}, url = {https://papers.ssrn.com/abstract=3046657}, language = {en}, number = {ID 3046657}, institution = {Social Science Research Network}, author = {Smalter Hall, Aaron and Cook, Thomas R.}, month = sep, year = {2017} }
    RIS
    IEEE
    [1]
    A. Smalter Hall and T. R. Cook, “Macroeconomic Indicator Forecasting with Deep Neural Networks,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3046657, Sep. 2017.
  4. ⊞ Neural Decomposition of Time-Series Data for Effective Generalization IEEE Transactions on Neural Networks and Learning Systems (2017)
    [1]
    L. B. Godfrey and M. S. Gashler, “Neural Decomposition of Time-Series Data for Effective Generalization,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–13, 2017.
    00000
    IEEE Transactions on Neural Networks and Learning Systems
    Abstract
    We present a neural network technique for the analysis and extrapolation of time-series data called Neural Decomposition (ND). Units with a sinusoidal activation function are used to perform a Fourier-like decomposition of training samples into a sum of sinusoids‚ augmented by units with nonperiodic activation functions to capture linear trends and other nonperiodic components. We show how careful weight initialization can be combined with regularization to form a simple model that generalizes well. Our method generalizes effectively on the Mackey-Glass series‚ a dataset of unemployment rates as reported by the U.S. Department of Labor Statistics‚ a time-series of monthly international airline passengers‚ the monthly ozone concentration in downtown Los Angeles‚ and an unevenly sampled time-series of oxygen isotope measurements from a cave in north India. We find that ND outperforms popular time-series forecasting techniques including LSTM‚ echo state networks‚ ARIMA‚ SARIMA‚ SVR with a radial basis function‚ and Gashler and Ashmore’s model.
    link
    BIB
    @article{godfrey2017neural, title = {Neural {Decomposition} of {Time}-{Series} {Data} for {Effective} {Generalization}}, issn = {2162-237X, 2162-2388}, url = {http://arxiv.org/abs/1705.09137}, doi = {10.1109/TNNLS.2017.2709324}, journal = {IEEE Transactions on Neural Networks and Learning Systems}, author = {Godfrey, Luke B. and Gashler, Michael S.}, year = {2017}, note = {00000}, pages = {1--13} }
    RIS
    IEEE
    [1]
    L. B. Godfrey and M. S. Gashler, “Neural Decomposition of Time-Series Data for Effective Generalization,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–13, 2017.
  5. ⊞ Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks (2017)
    [1]
    P. R. Lorenzo, J. Nalepa, M. Kawulok, L. S. Ramos, and J. R. Pastor, “Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks,” in Proceedings of the Genetic and Evolutionary Computation Conference, New York, NY, USA, 2017, pp. 481–488.
    00000
    GECCO '17
    Abstract
    Deep neural networks (DNNs) have achieved unprecedented success in a wide array of tasks. However‚ the performance of these systems depends directly on their hyper-parameters which often must be selected by an expert. Optimizing the hyper-parameters remains a substantial obstacle in designing DNNs in practice. In this work‚ we propose to select them using particle swarm optimization (PSO). Such biologically-inspired approaches have not been extensively exploited for this task. We demonstrate that PSO efficiently explores the solution space‚ allowing DNNs of a minimal topology to obtain competitive classification performance over the MNIST dataset. We showed that very small DNNs optimized by PSO retrieve promising classification accuracy for CIFAR-10. Also‚ PSO improves the performance of existing architectures. Extensive experimental study‚ backed-up with the statistical tests‚ revealed that PSO is an effective technique for automating hyper-parameter selection and efficiently exploits computational resources.
    link
    BIB
    @inproceedings{lorenzo2017particle, address = {New York, NY, USA}, series = {{GECCO} '17}, title = {Particle {Swarm} {Optimization} for {Hyper}-parameter {Selection} in {Deep} {Neural} {Networks}}, isbn = {978-1-4503-4920-8}, url = {http://doi.acm.org/10.1145/3071178.3071208}, doi = {10.1145/3071178.3071208}, booktitle = {Proceedings of the {Genetic} and {Evolutionary} {Computation} {Conference}}, publisher = {ACM}, author = {Lorenzo, Pablo Ribalta and Nalepa, Jakub and Kawulok, Michal and Ramos, Luciano Sanchez and Pastor, José Ranilla}, year = {2017}, note = {00000}, pages = {481--488} }
    RIS
    IEEE
    [1]
    P. R. Lorenzo, J. Nalepa, M. Kawulok, L. S. Ramos, and J. R. Pastor, “Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks,” in Proceedings of the Genetic and Evolutionary Computation Conference, New York, NY, USA, 2017, pp. 481–488.
  6. ⊞ Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization (2017)
    [1]
    P. R. Lorenzo, J. Nalepa, L. S. Ramos, and J. R. Pastor, “Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, New York, NY, USA, 2017, pp. 1864–1871.
    GECCO '17
    Abstract
    The need of manual hyper-parameter selection can seriously hamper the model optimization of Deep Neural Networks (DNNs). Conventional automated approaches tackling this problem suffer from poor scalability or fail in certain scenarios. In this paper‚ we introduce a parallel method that applies Particle Swarm Optimization (PSO) for the hyper-parameter selection in DNNs. To estimate the best hyper-parameters‚ a population of particles is evolved‚ with their fitness calculated in parallel. The experimental results demonstrate very desirable scalability properties for different DNNs. We show that the parallel PSO can further optimize existent models designed by experts in an affordable amount of time.
    link
    BIB
    @inproceedings{lorenzo2017hyper-parameter, address = {New York, NY, USA}, series = {{GECCO} '17}, title = {Hyper-parameter {Selection} in {Deep} {Neural} {Networks} {Using} {Parallel} {Particle} {Swarm} {Optimization}}, isbn = {978-1-4503-4939-0}, url = {http://doi.acm.org/10.1145/3067695.3084211}, doi = {10.1145/3067695.3084211}, booktitle = {Proceedings of the {Genetic} and {Evolutionary} {Computation} {Conference} {Companion}}, publisher = {ACM}, author = {Lorenzo, Pablo Ribalta and Nalepa, Jakub and Ramos, Luciano Sanchez and Pastor, José Ranilla}, year = {2017}, pages = {1864--1871} }
    RIS
    IEEE
    [1]
    P. R. Lorenzo, J. Nalepa, L. S. Ramos, and J. R. Pastor, “Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, New York, NY, USA, 2017, pp. 1864–1871.
  7. ⊞ Evolving Deep Neural Networks arXiv:1703.00548 [cs] (March 2017)
    [1]
    R. Miikkulainen et al., “Evolving Deep Neural Networks,” arXiv:1703.00548 [cs], Mar. 2017.
    00058
    arXiv:1703.00548 [cs]
    Abstract
    The success of deep learning depends on finding an architecture to fit the task. As deep learning has scaled up to more challenging tasks‚ the architectures have become difficult to design by hand. This paper proposes an automated method‚ CoDeepNEAT‚ for optimizing deep learning architectures through evolution. By extending existing neuroevolution methods to topology‚ components‚ and hyperparameters‚ this method achieves results comparable to best human designs in standard benchmarks in object recognition and language modeling. It also supports building a real-world application of automated image captioning on a magazine website. Given the anticipated increases in available computing power‚ evolution of deep networks is promising approach to constructing deep learning applications in the future.
    link
    BIB
    @article{miikkulainen2017evolving, title = {Evolving {Deep} {Neural} {Networks}}, url = {http://arxiv.org/abs/1703.00548}, journal = {arXiv:1703.00548 [cs]}, author = {Miikkulainen, Risto and Liang, Jason and Meyerson, Elliot and Rawal, Aditya and Fink, Dan and Francon, Olivier and Raju, Bala and Shahrzad, Hormoz and Navruzyan, Arshak and Duffy, Nigel and Hodjat, Babak}, month = mar, year = {2017}, note = {00058} }
    RIS
    IEEE
    [1]
    R. Miikkulainen et al., “Evolving Deep Neural Networks,” arXiv:1703.00548 [cs], Mar. 2017.
  8. ⊞ Deep Learning (2016)
    [1]
    I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
    02804
    link
    BIB
    @book{goodfellow2016deep, title = {Deep {Learning}}, url = {http://www.deeplearningbook.org}, publisher = {MIT Press}, author = {Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron}, year = {2016}, note = {02804} }
    RIS
    IEEE
    [1]
    I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
  9. ⊞ Learning Ordinary Differential Equations for Macroeconomic Modelling (December 2015)
    [1]
    Z. Georgiev and D. Kazakov, “Learning Ordinary Differential Equations for Macroeconomic Modelling,” in 2015 IEEE Symposium Series on Computational Intelligence, 2015, pp. 905–909.
    Abstract
    This article describes an empirical approach to the macroeconomic modelling of the Euro zone. Data for the period 1971 – 2007 has been used to learn systems of ordinary differential equations (ODE) linking inflation‚ real interest and output growth. The equation discovery algorithm LAGRAMGE was used in conjunction with a grammar defining a potentially large range of possible parametric equations. The coefficients of each equation are automatically fitted on the training data and the ones with the lowest error rates returned as a result. We have added a tool for out-of-sample error evaluation to the in-sample evaluation built in LAGRAMGE. The paper compares the performance of ODE models to previous work on the learning of ordinary equations for the same purpose.
    BIB
    @inproceedings{georgiev2015learning, title = {Learning {Ordinary} {Differential} {Equations} for {Macroeconomic} {Modelling}}, doi = {10.1109/SSCI.2015.133}, booktitle = {2015 {IEEE} {Symposium} {Series} on {Computational} {Intelligence}}, author = {Georgiev, Z. and Kazakov, D.}, month = dec, year = {2015}, pages = {905--909} }
    RIS
    IEEE
    [1]
    Z. Georgiev and D. Kazakov, “Learning Ordinary Differential Equations for Macroeconomic Modelling,” in 2015 IEEE Symposium Series on Computational Intelligence, 2015, pp. 905–909.
  10. ⊞ Guide to NumPy: 2nd Edition (September 2015)
    [1]
    T. E. Oliphant PhD, Guide to NumPy: 2nd Edition, 2 edition. Austin, Tex.: CreateSpace Independent Publishing Platform, 2015.
    Abstract
    This is the second edition of Travis Oliphant’s A Guide to NumPy originally published electronically in 2006. It is designed to be a reference that can be used by practitioners who are familiar with Python but want to learn more about NumPy and related tools. In this updated edition‚ new perspectives are shared as well as descriptions of new distributed processing tools in the ecosystem‚ and how Numba can be used to compile code using NumPy arrays. Travis Oliphant is the co-founder and CEO of Continuum Analytics. Continuum Analytics develops Anaconda‚ the leading modern open source analytics platform powered by Python. Travis‚ who is a passionate advocate of open source technology‚ has a Ph.D. from Mayo Clinic and B.S. and M.S. degrees in Mathematics and Electrical Engineering from Brigham Young University. Since 1997‚ he has worked extensively with Python for computational and data science. He was the primary creator of the NumPy package and founding contributor to the SciPy package. He was also a co-founder and past board member of NumFOCUS‚ a non-profit for reproducible and accessible science that supports the PyData stack. He also served on the board of the Python Software Foundation.
    BIB
    @book{oliphant_phd_guide_2015, address = {Austin, Tex.}, edition = {2 edition}, title = {Guide to {NumPy}: 2nd {Edition}}, isbn = {978-1-5173-0007-4}, language = {English}, publisher = {CreateSpace Independent Publishing Platform}, author = {Oliphant PhD, Travis E.}, month = sep, year = {2015} }
    RIS
    IEEE
    [1]
    T. E. Oliphant PhD, Guide to NumPy: 2nd Edition, 2 edition. Austin, Tex.: CreateSpace Independent Publishing Platform, 2015.
  11. ⊞ TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015)
    [1]
    Martín Abadi et al., TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015.
    link
    BIB
    @book{martin_abadi_tensorflow:_2015, title = {{TensorFlow}: {Large}-{Scale} {Machine} {Learning} on {Heterogeneous} {Systems}}, url = {https://www.tensorflow.org/}, author = {{Martín Abadi} and {Ashish Agarwal} and {Paul Barham} and {Eugene Brevdo} and {Zhifeng Chen} and {Craig Citro} and {Greg S. Corrado} and {Andy Davis} and {Jeffrey Dean} and {Matthieu Devin} and {Sanjay Ghemawat} and {Ian Goodfellow} and {Andrew Harp} and {Geoffrey Irving} and {Michael Isard} and Jia, Yangqing and {Rafal Jozefowicz} and {Lukasz Kaiser} and {Manjunath Kudlur} and {Josh Levenberg} and {Dandelion Mané} and {Rajat Monga} and {Sherry Moore} and {Derek Murray} and {Chris Olah} and {Mike Schuster} and {Jonathon Shlens} and {Benoit Steiner} and {Ilya Sutskever} and {Kunal Talwar} and {Paul Tucker} and {Vincent Vanhoucke} and {Vijay Vasudevan} and {Fernanda Viégas} and {Oriol Vinyals} and {Pete Warden} and {Martin Wattenberg} and {Martin Wicke} and {Yuan Yu} and {Xiaoqiang Zheng}}, year = {2015} }
    RIS
    IEEE
    [1]
    Martín Abadi et al., TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015.
  12. ⊞ Keras (2015)
    [1]
    F. Chollet and others, Keras. 2015.
    00000
    link
    BIB
    @book{chollet2015keras, title = {Keras}, url = {https://keras.io}, author = {Chollet, François and {others}}, year = {2015}, note = {00000} }
    RIS
    IEEE
    [1]
    F. Chollet and others, Keras. 2015.
  13. ⊞ Time Series Analysis: Forecasting and Control (May 2015)
    [1]
    G. E. P. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time Series Analysis: Forecasting and Control. John Wiley & Sons, 2015.
    Abstract
    Praise for the Fourth Edition “The book follows faithfully the style of the original edition. The approach is heavily motivated by real-world time series‚ and by developing a complete approach to model building‚ estimation‚ forecasting and control." - Mathematical Reviews Bridging classical models and modern topics‚ the Fifth Edition of Time Series Analysis: Forecasting and Control maintains a balanced presentation of the tools for modeling and analyzing time series. Also describing the latest developments that have occurred in the field over the past decade through applications from areas such as business‚ finance‚ and engineering‚ the Fifth Edition continues to serve as one of the most influential and prominent works on the subject. Time Series Analysis: Forecasting and Control‚ Fifth Edition provides a clearly written exploration of the key methods for building‚ classifying‚ testing‚ and analyzing stochastic models for time series and describes their use in five important areas of application: forecasting; determining the transfer function of a system; modeling the effects of intervention events; developing multivariate dynamic models; and designing simple control schemes. Along with these classical uses‚ the new edition covers modern topics with new features that include: A redesigned chapter on multivariate time series analysis with an expanded treatment of Vector Autoregressive‚ or VAR models‚ along with a discussion of the analytical tools needed for modeling vector time series An expanded chapter on special topics covering unit root testing‚ time-varying volatility models such as ARCH and GARCH‚ nonlinear time series models‚ and long memory models Numerous examples drawn from finance‚ economics‚ engineering‚ and other related fields The use of the publicly available R software for graphical illustrations and numerical calculations along with scripts that demonstrate the use of R for model building and forecasting Updates to literature references throughout and new end-of-chapter exercises Streamlined chapter introductions and revisions that update and enhance the exposition Time Series Analysis: Forecasting and Control‚ Fifth Edition is a valuable real-world reference for researchers and practitioners in time series analysis‚ econometrics‚ finance‚ and related fields. The book is also an excellent textbook for beginning graduate-level courses in advanced statistics‚ mathematics‚ economics‚ finance‚ engineering‚ and physics.
    BIB
    @book{box2015time, title = {Time {Series} {Analysis}: {Forecasting} and {Control}}, isbn = {978-1-118-67492-5}, language = {en}, publisher = {John Wiley \& Sons}, author = {Box, George E. P. and Jenkins, Gwilym M. and Reinsel, Gregory C. and Ljung, Greta M.}, month = may, year = {2015} }
    RIS
    IEEE
    [1]
    G. E. P. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time Series Analysis: Forecasting and Control. John Wiley & Sons, 2015.
  14. ⊞ Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification arXiv:1502.01852 [cs] (February 2015)
    [1]
    K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” arXiv:1502.01852 [cs], Feb. 2015.
    arXiv:1502.01852 [cs]
    Abstract
    Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work‚ we study rectifier neural networks for image classification from two aspects. First‚ we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second‚ we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets)‚ we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet‚ 6.66%). To our knowledge‚ our result is the first to surpass human-level performance (5.1%‚ Russakovsky et al.) on this visual recognition challenge.
    link
    BIB
    @article{he2015delving, title = {Delving {Deep} into {Rectifiers}: {Surpassing} {Human}-{Level} {Performance} on {ImageNet} {Classification}}, url = {http://arxiv.org/abs/1502.01852}, journal = {arXiv:1502.01852 [cs]}, author = {He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian}, month = feb, year = {2015} }
    RIS
    IEEE
    [1]
    K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” arXiv:1502.01852 [cs], Feb. 2015.
  15. ⊞ Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift arXiv:1502.03167 [cs] (February 2015)
    [1]
    S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv:1502.03167 [cs], Feb. 2015.
    04808
    arXiv:1502.03167 [cs]
    Abstract
    Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training‚ as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization‚ and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift‚ and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer‚ in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model‚ Batch Normalization achieves the same accuracy with 14 times fewer training steps‚ and beats the original model by a significant margin. Using an ensemble of batch-normalized networks‚ we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error)‚ exceeding the accuracy of human raters.
    link
    BIB
    @article{ioffe2015batch, title = {Batch {Normalization}: {Accelerating} {Deep} {Network} {Training} by {Reducing} {Internal} {Covariate} {Shift}}, url = {http://arxiv.org/abs/1502.03167}, journal = {arXiv:1502.03167 [cs]}, author = {Ioffe, Sergey and Szegedy, Christian}, month = feb, year = {2015}, note = {04808} }
    RIS
    IEEE
    [1]
    S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv:1502.03167 [cs], Feb. 2015.
  16. ⊞ Adam: A Method for Stochastic Optimization arXiv:1412.6980 [cs] (December 2014)
    [1]
    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980 [cs], Dec. 2014.
    arXiv:1412.6980 [cs]
    Abstract
    We introduce Adam‚ an algorithm for first-order gradient-based optimization of stochastic objective functions‚ based on adaptive estimates of lower-order moments. The method is straightforward to implement‚ is computationally efficient‚ has little memory requirements‚ is invariant to diagonal rescaling of the gradients‚ and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms‚ on which Adam was inspired‚ are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally‚ we discuss AdaMax‚ a variant of Adam based on the infinity norm.
    link
    BIB
    @article{kingma2014adam:, title = {Adam: {A} {Method} for {Stochastic} {Optimization}}, url = {http://arxiv.org/abs/1412.6980}, journal = {arXiv:1412.6980 [cs]}, author = {Kingma, Diederik P. and Ba, Jimmy}, month = dec, year = {2014} }
    RIS
    IEEE
    [1]
    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980 [cs], Dec. 2014.
  17. ⊞ seaborn: v0.5.0 (November 2014) (November 2014)
    [1]
    M. Waskom et al., seaborn: v0.5.0 (November 2014). Zenodo, 2014.
    DOI: 10.5281/zenodo.12710
    Abstract
    This is a major release from 0.4. Highlights include new functions for plotting heatmaps‚ possibly while applying clustering algorithms to discover structured relationships. These functions are complemented by new custom colormap functions and a full set of IPython widgets that allow interactive selection of colormap parameters. The palette tutorial has been rewritten to cover these new tools and more generally provide guidance on how to use color in visualizations. There are also a number of smaller changes and bugfixes. Plotting functions Added the heatmap function for visualizing a matrix of data by color-encoding the values. See the docs for more information. Added the clustermap function for clustering and visualizing a matrix of data‚ with options to label individual rows and columns by colors. See the docs for more information. This work was lead by Olga Botvinnik. lmplot and pairplot get a new keyword argument‚ markers. This can be a single kind of marker or a list of different markers for each level of the hue variable. Using different markers for different hues should let plots be more comprehensible when reproduced to black-and-white (i.e. when printed). See the github pull request for examples. More generally‚ there is a new keyword argument in FacetGrid and PairGrid‚ hue_kws. This similarly lets plot aesthetics vary across the levels of the hue variable‚ but more flexibily. hue_kws should be a dictionary that maps the name of keyword arguments to lists of values that are as long as the number of levels of the hue variable. The argument subplot_kws has been added to FacetGrid. This allows for faceted plots with custom projections‚ including maps with Cartopy. Color palettes Added two new functions to create custom color palettes. For sequential palettes‚ you can use the light_palette function‚ which takes a seed color and creates a ramp from a very light‚ desaturated variant of it. For diverging palettes‚ you can use the diverging_palette function to create a balanced ramp between two endpoints to a light or dark midpoint. See the palette tutorial \textbackslashtextlesspalette_tutorial\textbackslashtextgreater for more information. Added the ability to specify the seed color for light_palette and dark_palette as a tuple of husl or hls space values or as a named xkcd color. The interpretation of the seed color is now provided by the new input parameter to these functions. Added several new interactive palette widgets: choose_colorbrewer_palette‚ choose_light_palette‚ choose_dark_palette‚ and choose_diverging_palette. For consistency‚ renamed the cubehelix widget to choose_cubehelix_palette (and fixed a bug where the cubehelix palette was reversed). These functions also now return either a color palette list or a matplotlib colormap when called‚ and that object will be live-updated as you play with the widget. This should make it easy to iterate over a plot until you find a good representation for the data. See the Github pull request or this notebook (download it to use the widgets) for more information. Overhauled the color palette tutorial to organize the discussion by class of color palette and provide more motivation behind the various choices one might make when choosing colors for their data. Bug fixes Fixed a bug in PairGrid that gave incorrect results (or a crash) when the input DataFrame has a non-default index. Fixed a bug in PairGrid where passing columns with a date-like datatype raised an exception. Fixed a bug where lmplot would show a legend when the hue variable was also used on either the rows or columns (making the legend redundant). Worked around a matplotlib bug that was forcing outliers in boxplot to appear as blue. kdeplot now accepts pandas Series for the data and data2 arguments. Using a non-default correlation method in corrplot now implies sig_stars=False as the permutation test used to significance values for the correlations uses a pearson metric. Removed pdf.fonttype from the style definitions‚ as the value used in version 0.4 resulted in very large PDF files.
    link
    BIB
    @book{waskom2014seaborn:, title = {seaborn: v0.5.0 ({November} 2014)}, url = {https://zenodo.org/record/12710#.Ww_VRHXwaV4}, publisher = {Zenodo}, author = {Waskom, Michael and Botvinnik, Olga and Hobson, Paul and Cole, John B. and Halchenko, Yaroslav and Hoyer, Stephan and Miles, Alistair and Augspurger, Tom and Yarkoni, Tal and Megies, Tobias and Coelho, Luis Pedro and Wehner, Daniel and {cynddl} and Ziegler, Erik and {diego0020} and Zaytsev, Yury V. and Hoppe, Travis and Seabold, Skipper and Cloud, Phillip and Koskinen, Miikka and Meyer, Kyle and Qalieh, Adel and Allan, Dan}, month = nov, year = {2014}, doi = {10.5281/zenodo.12710} }
    RIS
    IEEE
    [1]
    M. Waskom et al., seaborn: v0.5.0 (November 2014). Zenodo, 2014.
  18. ⊞ Once you SCOOP, no need to fork (2014)
    [1]
    Y. Hold-Geoffroy, O. Gagnon, and M. Parizeau, “Once you SCOOP, no need to fork,” in Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment, 2014, p. 60.
    BIB
    @inproceedings{hold-geoffroy2014once, title = {Once you {SCOOP}, no need to fork}, booktitle = {Proceedings of the 2014 {Annual} {Conference} on {Extreme} {Science} and {Engineering} {Discovery} {Environment}}, publisher = {ACM}, author = {Hold-Geoffroy, Yannick and Gagnon, Olivier and Parizeau, Marc}, year = {2014}, pages = {60} }
    RIS
    IEEE
    [1]
    Y. Hold-Geoffroy, O. Gagnon, and M. Parizeau, “Once you SCOOP, no need to fork,” in Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment, 2014, p. 60.
  19. ⊞ Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014)
    [1]
    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” p. 30, 2014.
    Abstract
    Deep neural nets with a large number of parameters are very powerful machine learning systems. However‚ overfitting is a serious problem in such networks. Large networks are also slow to use‚ making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training‚ dropout samples from an exponential number of different “thinned” networks. At test time‚ it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision‚ speech recognition‚ document classification and computational biology‚ obtaining state-of-the-art results on many benchmark data sets.
    BIB
    @article{srivastava2014dropout:, title = {Dropout: {A} {Simple} {Way} to {Prevent} {Neural} {Networks} from {Overfitting}}, language = {en}, author = {Srivastava, Nitish and Hinton, Geoffrey and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan}, year = {2014}, pages = {30} }
    RIS
    IEEE
    [1]
    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” p. 30, 2014.
  20. ⊞ API design for machine learning software: experiences from the scikit-learn project (2013)
    [1]
    L. Buitinck et al., “API design for machine learning software: experiences from the scikit-learn project,” in ECML PKDD Workshop: Languages for Data Mining and Machine Learning, 2013, pp. 108–122.
    BIB
    @inproceedings{buitinck2013api, title = {{API} design for machine learning software: experiences from the scikit-learn project}, booktitle = {{ECML} {PKDD} {Workshop}: {Languages} for {Data} {Mining} and {Machine} {Learning}}, author = {Buitinck, Lars and Louppe, Gilles and Blondel, Mathieu and Pedregosa, Fabian and Mueller, Andreas and Grisel, Olivier and Niculae, Vlad and Prettenhofer, Peter and Gramfort, Alexandre and Grobler, Jaques and Layton, Robert and VanderPlas, Jake and Joly, Arnaud and Holt, Brian and Varoquaux, Gaël}, year = {2013}, pages = {108--122} }
    RIS
    IEEE
    [1]
    L. Buitinck et al., “API design for machine learning software: experiences from the scikit-learn project,” in ECML PKDD Workshop: Languages for Data Mining and Machine Learning, 2013, pp. 108–122.
  21. ⊞ Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms (2013)
    [1]
    C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms,” in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2013, pp. 847–855.
    00354
    KDD '13
    Abstract
    Many different machine learning algorithms exist; taking into account each algorithm’s hyperparameters‚ there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters‚ going beyond previous work that attacks these issues separately. We show that this problem can be addressed by a fully automated approach‚ leveraging recent innovations in Bayesian optimization. Specifically‚ we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA’s standard distribution‚ spanning 2 ensemble methods‚ 10 meta-methods‚ 27 base classifiers‚ and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository‚ the KDD Cup 09‚ variants of the MNIST dataset and CIFAR-10‚ we show classification performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications‚ and hence to achieve improved performance.
    link
    BIB
    @inproceedings{thornton2013auto-weka:, address = {New York, NY, USA}, series = {{KDD} '13}, title = {Auto-{WEKA}: {Combined} {Selection} and {Hyperparameter} {Optimization} of {Classification} {Algorithms}}, isbn = {978-1-4503-2174-7}, url = {http://doi.acm.org/10.1145/2487575.2487629}, doi = {10.1145/2487575.2487629}, booktitle = {Proceedings of the 19th {ACM} {SIGKDD} {International} {Conference} on {Knowledge} {Discovery} and {Data} {Mining}}, publisher = {ACM}, author = {Thornton, Chris and Hutter, Frank and Hoos, Holger H. and Leyton-Brown, Kevin}, year = {2013}, note = {00354}, pages = {847--855} }
    RIS
    IEEE
    [1]
    C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms,” in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2013, pp. 847–855.
  22. ⊞ ADADELTA: An Adaptive Learning Rate Method arXiv:1212.5701 [cs] (December 2012)
    [1]
    M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” arXiv:1212.5701 [cs], Dec. 2012.
    01742
    arXiv:1212.5701 [cs]
    Abstract
    We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information‚ different model architecture choices‚ various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
    link
    BIB
    @article{zeiler2012adadelta:, title = {{ADADELTA}: {An} {Adaptive} {Learning} {Rate} {Method}}, url = {http://arxiv.org/abs/1212.5701}, journal = {arXiv:1212.5701 [cs]}, author = {Zeiler, Matthew D.}, month = dec, year = {2012}, note = {01742} }
    RIS
    IEEE
    [1]
    M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” arXiv:1212.5701 [cs], Dec. 2012.
  23. ⊞ On the difficulty of training Recurrent Neural Networks arXiv:1211.5063 [cs] (November 2012)
    [1]
    R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training Recurrent Neural Networks,” arXiv:1211.5063 [cs], Nov. 2012.
    01016
    arXiv:1211.5063 [cs]
    Abstract
    Training Recurrent Neural Networks is more troublesome than feedforward ones because of the vanishing and exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to understand the fundamental issues underlying the exploding gradient problem by exploring it from an analytical‚ a geometric and a dynamical system perspective. Our analysis is used to justify the simple yet effective solution of norm clipping the exploded gradient. In the experimental section‚ the comparison between this heuristic solution and standard SGD provides empirical evidence towards our hypothesis as well as it shows that such a heuristic is required to reach state of the art results on a character prediction task and a polyphonic music prediction one.
    link
    BIB
    @article{pascanu2012difficulty, title = {On the difficulty of training {Recurrent} {Neural} {Networks}}, url = {http://arxiv.org/abs/1211.5063}, language = {en}, journal = {arXiv:1211.5063 [cs]}, author = {Pascanu, Razvan and Mikolov, Tomas and Bengio, Yoshua}, month = nov, year = {2012}, note = {01016} }
    RIS
    IEEE
    [1]
    R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training Recurrent Neural Networks,” arXiv:1211.5063 [cs], Nov. 2012.
  24. ⊞ Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude COURSERA: Neural networks for machine learning (2012)
    [1]
    T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning, vol. 4, no. 2, pp. 26–31, 2012.
    01262
    COURSERA: Neural networks for machine learning
    BIB
    @article{tieleman2012lecture, title = {Lecture 6.5-rmsprop: {Divide} the gradient by a running average of its recent magnitude}, volume = {4}, number = {2}, journal = {COURSERA: Neural networks for machine learning}, author = {Tieleman, Tijmen and Hinton, Geoffrey}, year = {2012}, note = {01262}, pages = {26--31} }
    RIS
    IEEE
    [1]
    T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning, vol. 4, no. 2, pp. 26–31, 2012.
  25. ⊞ Sequential Model-Based Optimization for General Algorithm Configuration (2011)
    [1]
    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential Model-Based Optimization for General Algorithm Configuration,” in Learning and Intelligent Optimization, vol. 6683, C. A. C. Coello, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 507–523.
    DOI: 10.1007/978-3-642-25566-3_40
    Abstract
    State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance. However‚ manually exploring the resulting combinatorial space of parameter settings is tedious and tends to lead to unsatisfactory outcomes. Recently‚ automated approaches for solving this algorithm configuration problem have led to substantial improvements in the state of the art for solving various problems. One promising approach constructs explicit regression models to describe the dependence of target algorithm performance on parameter settings; however‚ this approach has so far been limited to the optimization of few numerical algorithm parameters on single instances. In this paper‚ we extend this paradigm for the first time to general algorithm configuration problems‚ allowing many categorical parameters and optimization for sets of instances. We experimentally validate our new algorithm configuration procedure by optimizing a local search and a tree search solver for the propositional satisfiability problem (SAT)‚ as well as the commercial mixed integer programming (MIP) solver CPLEX. In these experiments‚ our procedure yielded state-of-the-art performance‚ and in many cases outperformed the previous best configuration approach.
    link
    BIB
    @incollection{hutter2011sequential, address = {Berlin, Heidelberg}, title = {Sequential {Model}-{Based} {Optimization} for {General} {Algorithm} {Configuration}}, volume = {6683}, isbn = {978-3-642-25565-6 978-3-642-25566-3}, url = {http://link.springer.com/10.1007/978-3-642-25566-3_40}, language = {en}, booktitle = {Learning and {Intelligent} {Optimization}}, publisher = {Springer Berlin Heidelberg}, author = {Hutter, Frank and Hoos, Holger H. and Leyton-Brown, Kevin}, editor = {Coello, Carlos A. Coello}, year = {2011}, doi = {10.1007/978-3-642-25566-3_40}, pages = {507--523} }
    RIS
    IEEE
    [1]
    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential Model-Based Optimization for General Algorithm Configuration,” in Learning and Intelligent Optimization, vol. 6683, C. A. C. Coello, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 507–523.
  26. ⊞ Deep Sparse Rectifier Neural Networks (2011)
    [1]
    X. Glorot, A. Bordes, and Y. Bengio, “Deep Sparse Rectifier Neural Networks,” p. 9, 2011.
    Abstract
    While logistic sigmoid neurons are more biologically plausible than hyperbolic tangent neurons‚ the latter work better for training multi-layer neural networks. This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-differentiability at zero‚ creating sparse representations with true zeros‚ which seem remarkably suitable for naturally sparse data. Even though they can take advantage of semi-supervised setups with extra-unlabeled data‚ deep rectifier networks can reach their best performance without requiring any unsupervised pre-training on purely supervised tasks with large labeled datasets. Hence‚ these results can be seen as a new milestone in the attempts at understanding the difficulty in training deep but purely supervised neural networks‚ and closing the performance gap between neural networks learnt with and without unsupervised pre-training.
    BIB
    @article{glorot2011deep, title = {Deep {Sparse} {Rectifier} {Neural} {Networks}}, language = {en}, author = {Glorot, Xavier and Bordes, Antoine and Bengio, Yoshua}, year = {2011}, pages = {9} }
    RIS
    IEEE
    [1]
    X. Glorot, A. Bordes, and Y. Bengio, “Deep Sparse Rectifier Neural Networks,” p. 9, 2011.
  27. ⊞ Adaptive Subgradient Methods for Online Learning and Stochastic Optimization Journal of Machine Learning Research (2011)
    [1]
    J. Duchi, E. Hazan, and Y. Singer, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,” Journal of Machine Learning Research, vol. 12, no. Jul, pp. 2121–2159, 2011.
    Journal of Machine Learning Research
    link
    BIB
    @article{duchi2011adaptive, title = {Adaptive {Subgradient} {Methods} for {Online} {Learning} and {Stochastic} {Optimization}}, volume = {12}, issn = {ISSN 1533-7928}, url = {http://www.jmlr.org/papers/v12/duchi11a.html}, number = {Jul}, journal = {Journal of Machine Learning Research}, author = {Duchi, John and Hazan, Elad and Singer, Yoram}, year = {2011}, pages = {2121--2159} }
    RIS
    IEEE
    [1]
    J. Duchi, E. Hazan, and Y. Singer, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,” Journal of Machine Learning Research, vol. 12, no. Jul, pp. 2121–2159, 2011.
  28. ⊞ Akaike's Information Criterion (2011)
    [1]
    H. Akaike, “Akaike’s Information Criterion,” in International Encyclopedia of Statistical Science, Springer, Berlin, Heidelberg, 2011, pp. 25–25.
    00093 DOI: 10.1007/978-3-642-04898-2_110
    link
    BIB
    @incollection{akaike2011akaikes, title = {Akaike's {Information} {Criterion}}, url = {https://link.springer.com/referenceworkentry/10.1007/978-3- 642-04898-2_110}, language = {en}, booktitle = {International {Encyclopedia} of {Statistical} {Science}}, publisher = {Springer, Berlin, Heidelberg}, author = {Akaike, Hirotugu}, year = {2011}, doi = {10.1007/978-3-642-04898-2_110}, note = {00093 }, pages = {25--25} }
    RIS
    IEEE
    [1]
    H. Akaike, “Akaike’s Information Criterion,” in International Encyclopedia of Statistical Science, Springer, Berlin, Heidelberg, 2011, pp. 25–25.
  29. ⊞ The effects of variable stationarity in a financial time-series on Artificial Neural Networks (April 2011)
    [1]
    M. Butler and D. Kazakov, “The effects of variable stationarity in a financial time-series on Artificial Neural Networks,” in 2011 IEEE Symposium on Computational Intelligence for Financial Engineering and Economics (CIFEr), 2011, pp. 1–8.
    00000
    Abstract
    This study investigates the characteristic of non-stationarity in a financial time-series and its effect on the learning process for Artificial Neural Networks (ANN). It is motivated by previous work where it was shown that non-stationarity is not static within a financial time series but quite variable in nature. Initially unit-root tests were performed to isolate segments that were stationary or non-stationary at a pre-determined significance level and then various tests were conducted based on forecasting accuracy. The hypothesis of this research is that when using the de-trended/original observations from the time series the trend/level stationary segments should produce lower error measures and when the series are differenced the difference stationary (non-stationary) segments should have lower error. The results to date reveal that the effects of variable stationarity on learning with ANNs are a function of forecasting time-horizon‚ strength of the linear-time trend‚ sample size and persistence of the stationary process.
    BIB
    @inproceedings{butler2011effects, title = {The effects of variable stationarity in a financial time-series on {Artificial} {Neural} {Networks}}, doi = {10.1109/CIFER.2011.5953557}, booktitle = {2011 {IEEE} {Symposium} on {Computational} {Intelligence} for {Financial} {Engineering} and {Economics} ({CIFEr})}, author = {Butler, M. and Kazakov, D.}, month = apr, year = {2011}, note = {00000}, pages = {1--8} }
    RIS
    IEEE
    [1]
    M. Butler and D. Kazakov, “The effects of variable stationarity in a financial time-series on Artificial Neural Networks,” in 2011 IEEE Symposium on Computational Intelligence for Financial Engineering and Economics (CIFEr), 2011, pp. 1–8.
  30. ⊞ A Survey on Transfer Learning IEEE Transactions on Knowledge and Data Engineering (October 2010)
    [1]
    S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
    IEEE Transactions on Knowledge and Data Engineering
    Abstract
    A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However‚ in many real-world applications‚ this assumption may not hold. For example‚ we sometimes have a classification task in one domain of interest‚ but we only have sufficient training data in another domain of interest‚ where the latter data may be in a different feature space or follow a different data distribution. In such cases‚ knowledge transfer‚ if done successfully‚ would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. In recent years‚ transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification‚ regression‚ and clustering problems. In this survey‚ we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation‚ multitask learning and sample selection bias‚ as well as covariate shift. We also explore some potential future issues in transfer learning research.
    BIB
    @article{pan2010survey, title = {A {Survey} on {Transfer} {Learning}}, volume = {22}, issn = {1041-4347}, doi = {10.1109/TKDE.2009.191}, number = {10}, journal = {IEEE Transactions on Knowledge and Data Engineering}, author = {Pan, S. J. and Yang, Q.}, month = oct, year = {2010}, pages = {1345--1359} }
    RIS
    IEEE
    [1]
    S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
  31. ⊞ Statsmodels: Econometric and statistical modeling with python (2010)
    [1]
    S. Seabold and J. Perktold, “Statsmodels: Econometric and statistical modeling with python,” in 9th Python in Science Conference, 2010.
    00000
    BIB
    @inproceedings{seabold2010statsmodels:, title = {Statsmodels: {Econometric} and statistical modeling with python}, booktitle = {9th {Python} in {Science} {Conference}}, author = {Seabold, Skipper and Perktold, Josef}, year = {2010}, note = {00000} }
    RIS
    IEEE
    [1]
    S. Seabold and J. Perktold, “Statsmodels: Econometric and statistical modeling with python,” in 9th Python in Science Conference, 2010.
  32. ⊞ Data Structures for Statistical Computing in Python (2010)
    [1]
    W. McKinney, “Data Structures for Statistical Computing in Python,” 2010, pp. 51–56.
    00000
    link
    BIB
    @inproceedings{mckinney2010data, title = {Data {Structures} for {Statistical} {Computing} in {Python}}, url = {http://conference.scipy.org/proceedings/scipy2010/ mckinney.html}, author = {McKinney, Wes}, year = {2010}, note = {00000}, pages = {51--56} }
    RIS
    IEEE
    [1]
    W. McKinney, “Data Structures for Statistical Computing in Python,” 2010, pp. 51–56.
  33. ⊞ Particle swarm optimization for parameter determination and feature selection of support vector machines Expert Systems with Applications (November 2008)
    [1]
    S.-W. Lin, K.-C. Ying, S.-C. Chen, and Z.-J. Lee, “Particle swarm optimization for parameter determination and feature selection of support vector machines,” Expert Systems with Applications, vol. 35, no. 4, pp. 1817–1824, Nov. 2008.
    Expert Systems with Applications
    Abstract
    Support vector machine (SVM) is a popular pattern classification method with many diverse applications. Kernel parameter setting in the SVM training procedure‚ along with the feature selection‚ significantly influences the classification accuracy. This study simultaneously determines the parameter values while discovering a subset of features‚ without reducing SVM classification accuracy. A particle swarm optimization (PSO) based approach for parameter determination and feature selection of the SVM‚ termed PSO+SVM‚ is developed. Several public datasets are employed to calculate the classification accuracy rate in order to evaluate the developed PSO+SVM approach. The developed approach was compared with grid search‚ which is a conventional method of searching parameter values‚ and other approaches. Experimental results demonstrate that the classification accuracy rates of the developed approach surpass those of grid search and many other approaches‚ and that the developed PSO+SVM approach has a similar result to GA+SVM. Therefore‚ the PSO+SVM approach is valuable for parameter determination and feature selection in an SVM.
    link
    BIB
    @article{lin2008particle, title = {Particle swarm optimization for parameter determination and feature selection of support vector machines}, volume = {35}, issn = {0957-4174}, url = {http://www.sciencedirect.com/science/article/pii/ S0957417407003752}, doi = {10.1016/j.eswa.2007.08.088}, number = {4}, journal = {Expert Systems with Applications}, author = {Lin, Shih-Wei and Ying, Kuo-Ching and Chen, Shih-Chieh and Lee, Zne-Jung}, month = nov, year = {2008}, pages = {1817--1824} }
    RIS
    IEEE
    [1]
    S.-W. Lin, K.-C. Ying, S.-C. Chen, and Z.-J. Lee, “Particle swarm optimization for parameter determination and feature selection of support vector machines,” Expert Systems with Applications, vol. 35, no. 4, pp. 1817–1824, Nov. 2008.
  34. ⊞ Equation discovery for macroeconomic modelling (2008)
    [1]
    D. Kazakov and T. Tsenova, “Equation discovery for macroeconomic modelling,” 2008.
    BIB
    @article{kazakov2008equation, title = {Equation discovery for macroeconomic modelling}, author = {Kazakov, Dimitar and Tsenova, Tsvetomira}, year = {2008} }
    RIS
    IEEE
    [1]
    D. Kazakov and T. Tsenova, “Equation discovery for macroeconomic modelling,” 2008.
  35. ⊞ Bayesian Information Criteria (2008)
    [1]
    “Bayesian Information Criteria,” in Information Criteria and Statistical Modeling, Springer, New York, NY, 2008, pp. 211–237.
    00640 DOI: 10.1007/978-0-387-71887-3_9
    Abstract
    This chapter considers model selection and evaluation criteria from a Bayesian point of view. A general framework for constructing the Bayesian information criterion (BIC) is described. The BIC is also extended such that it can be applied to the evaluation of models estimated by regularization. Section 9.2 presents Akaike’s Bayesian information criterion (ABIC) developed for the evaluation of Bayesian models having prior distributions with hyperparameters. In the latter half of this chapter‚ we consider information criteria for the evaluation of predictive distributions of Bayesian models. In particular‚ Section 9.3 gives examples of analytical evaluations of bias correction for linear Gaussian Bayes models. Section 9.4 describes‚ for general Bayesian models‚ how to estimate the asymptotic biases and how to perform the second-order bias correction by means of Laplace’s method for integrals.
    link
    BIB
    @incollection{noauthor2008bayesian, series = {Springer {Series} in {Statistics}}, title = {Bayesian {Information} {Criteria}}, isbn = {978-0-387-71886-6 978-0-387-71887-3}, url = {https://link.springer.com/chapter/10.1007/978-0-387-71887- 3_9}, language = {en}, booktitle = {Information {Criteria} and {Statistical} {Modeling}}, publisher = {Springer, New York, NY}, year = {2008}, doi = {10.1007/978-0-387-71887-3_9}, note = {00640 }, pages = {211--237} }
    RIS
    IEEE
    [1]
    “Bayesian Information Criteria,” in Information Criteria and Statistical Modeling, Springer, New York, NY, 2008, pp. 211–237.
  36. ⊞ Automatic Time Series Forecasting: The forecast Package for R Journal of Statistical Software, Articles (2008)
    [1]
    R. Hyndman and Y. Khandakar, “Automatic Time Series Forecasting: The forecast Package for R,” Journal of Statistical Software, Articles, vol. 27, no. 3, pp. 1–22, 2008.
    00002
    Journal of Statistical Software, Articles
    Abstract
    Automatic forecasts of large numbers of univariate time series are often needed in business and other contexts. We describe two automatic forecasting algorithms that have been implemented in the forecast package for R. The first is based on innovations state space models that underly exponential smoothing methods. The second is a step-wise algorithm for forecasting with ARIMA models. The algorithms are applicable to both seasonal and non-seasonal data‚ and are compared and illustrated using four real time series. We also briefly describe some of the other functionality available in the forecast package.
    link
    BIB
    @article{hyndman2008automatic, title = {Automatic {Time} {Series} {Forecasting}: {The} forecast {Package} for {R}}, volume = {27}, issn = {1548-7660}, url = {https://www.jstatsoft.org/v027/i03}, doi = {10.18637/jss.v027.i03}, number = {3}, journal = {Journal of Statistical Software, Articles}, author = {Hyndman, Rob and Khandakar, Yeasmin}, year = {2008}, note = {00002}, pages = {1--22} }
    RIS
    IEEE
    [1]
    R. Hyndman and Y. Khandakar, “Automatic Time Series Forecasting: The forecast Package for R,” Journal of Statistical Software, Articles, vol. 27, no. 3, pp. 1–22, 2008.
  37. ⊞ Quarterly Time-Series Forecasting With Neural Networks IEEE Transactions on Neural Networks (November 2007)
    [1]
    G. P. Zhang and D. M. Kline, “Quarterly Time-Series Forecasting With Neural Networks,” IEEE Transactions on Neural Networks, vol. 18, no. 6, pp. 1800–1814, Nov. 2007.
    IEEE Transactions on Neural Networks
    Abstract
    Forecasting of time series that have seasonal and other variations remains an important problem for forecasters. This paper presents a neural network (NN) approach to forecasting quarterly time series. With a large data set of 756 quarterly time series from the M3 forecasting competition‚ we conduct a comprehensive investigation of the effectiveness of several data preprocessing and modeling approaches. We consider two data preprocessing methods and 48 NN models with different possible combinations of lagged observations‚ seasonal dummy variables‚ trigonometric variables‚ and time index as inputs to the NN. Both parametric and nonparametric statistical analyses are performed to identify the best models under different circumstances and categorize similar models. Results indicate that simpler models‚ in general‚ outperform more complex models. In addition‚ data preprocessing especially with deseasonalization and detrending is very helpful in improving NN performance. Practical guidelines are also provided.
    BIB
    @article{zhang2007quarterly, title = {Quarterly {Time}-{Series} {Forecasting} {With} {Neural} {Networks}}, volume = {18}, issn = {1045-9227}, doi = {10.1109/TNN.2007.896859}, number = {6}, journal = {IEEE Transactions on Neural Networks}, author = {Zhang, G. P. and Kline, D. M.}, month = nov, year = {2007}, pages = {1800--1814} }
    RIS
    IEEE
    [1]
    G. P. Zhang and D. M. Kline, “Quarterly Time-Series Forecasting With Neural Networks,” IEEE Transactions on Neural Networks, vol. 18, no. 6, pp. 1800–1814, Nov. 2007.
  38. ⊞ Matplotlib: A 2D Graphics Environment Computing in Science & Engineering (May 2007)
    [1]
    J. D. Hunter, “Matplotlib: A 2D Graphics Environment,” Computing in Science & Engineering, vol. 9, no. 3, pp. 90–95, May 2007.
    00000
    Computing in Science & Engineering
    link
    BIB
    @article{hunter2007matplotlib:, title = {Matplotlib: {A} 2D {Graphics} {Environment}}, volume = {9}, issn = {1521-9615}, url = {https://aip.scitation.org/doi/abs/10.1109/MCSE.2007.55}, doi = {10.1109/MCSE.2007.55}, number = {3}, journal = {Computing in Science \& Engineering}, author = {Hunter, John D.}, month = may, year = {2007}, note = {00000}, pages = {90--95} }
    RIS
    IEEE
    [1]
    J. D. Hunter, “Matplotlib: A 2D Graphics Environment,” Computing in Science & Engineering, vol. 9, no. 3, pp. 90–95, May 2007.
  39. ⊞ 25 years of time series forecasting International Journal of Forecasting (January 2006)
    [1]
    J. G. De Gooijer and R. J. Hyndman, “25 years of time series forecasting,” International Journal of Forecasting, vol. 22, no. 3, pp. 443–473, Jan. 2006.
    International Journal of Forecasting
    Abstract
    We review the past 25 years of research into time series forecasting. In this silver jubilee issue‚ we naturally highlight results published in journals managed by the International Institute of Forecasters (Journal of Forecasting 1982–1985 and International Journal of Forecasting 1985–2005). During this period‚ over one third of all papers published in these journals concerned time series forecasting. We also review highly influential works on time series forecasting that have been published elsewhere during this period. Enormous progress has been made in many areas‚ but we find that there are a large number of topics in need of further development. We conclude with comments on possible future research directions in this field.
    link
    BIB
    @article{de_gooijer_25_2006, series = {Twenty five years of forecasting}, title = {25 years of time series forecasting}, volume = {22}, issn = {0169-2070}, url = {http://www.sciencedirect.com/science/article/pii/ S0169207006000021}, doi = {10.1016/j.ijforecast.2006.01.001}, number = {3}, journal = {International Journal of Forecasting}, author = {De Gooijer, Jan G. and Hyndman, Rob J.}, month = jan, year = {2006}, pages = {443--473} }
    RIS
    IEEE
    [1]
    J. G. De Gooijer and R. J. Hyndman, “25 years of time series forecasting,” International Journal of Forecasting, vol. 22, no. 3, pp. 443–473, Jan. 2006.
  40. ⊞ The Organization of Behavior: A Neuropsychological Theory (April 2005)
    [1]
    D. O. Hebb, The Organization of Behavior: A Neuropsychological Theory. Psychology Press, 2005.
    Abstract
    Since its publication in 1949‚ D.O. Hebb’s‚ The Organization of Behavior has been one of the most influential books in the fields of psychology and neuroscience. However‚ the original edition has been unavailable since 1966‚ ensuring that Hebb’s comment that a classic normally means "cited but not read" is true in his case. This new edition rectifies a long-standing problem for behavioral neuroscientists–the inability to obtain one of the most cited publications in the field. The Organization of Behavior played a significant part in stimulating the investigation of the neural foundations of behavior and continues to be inspiring because it provides a general framework for relating behavior to synaptic organization through the dynamics of neural networks. D.O. Hebb was also the first to examine the mechanisms by which environment and experience can influence brain structure and function‚ and his ideas formed the basis for work on enriched environments as stimulants for behavioral development. References to Hebb‚ the Hebbian cell assembly‚ the Hebb synapse‚ and the Hebb rule increase each year. These forceful ideas of 1949 are now applied in engineering‚ robotics‚ and computer science‚ as well as neurophysiology‚ neuroscience‚ and psychology–a tribute to Hebb’s foresight in developing a foundational neuropsychological theory of the organization of behavior.
    BIB
    @book{hebb2005organization, title = {The {Organization} of {Behavior}: {A} {Neuropsychological} {Theory}}, isbn = {978-1-135-63191-8}, language = {en}, publisher = {Psychology Press}, author = {Hebb, D. O.}, month = apr, year = {2005} }
    RIS
    IEEE
    [1]
    D. O. Hebb, The Organization of Behavior: A Neuropsychological Theory. Psychology Press, 2005.
  41. ⊞ Neural network forecasting for seasonal and trend time series European Journal of Operational Research (January 2005)
    [1]
    G. P. Zhang and M. Qi, “Neural network forecasting for seasonal and trend time series,” European Journal of Operational Research, vol. 160, no. 2, pp. 501–514, Jan. 2005.
    European Journal of Operational Research
    Abstract
    Neural networks have been widely used as a promising method for time series forecasting. However‚ limited empirical studies on seasonal time series forecasting with neural networks yield mixed results. While some find that neural networks are able to model seasonality directly and prior deseasonalization is not necessary‚ others conclude just the opposite. In this paper‚ we investigate the issue of how to effectively model time series with both seasonal and trend patterns. In particular‚ we study the effectiveness of data preprocessing‚ including deseasonalization and detrending‚ on neural network modeling and forecasting performance. Both simulation and real data are examined and results are compared to those obtained from the Box–Jenkins seasonal autoregressive integrated moving average models. We find that neural networks are not able to capture seasonal or trend variations effectively with the unpreprocessed raw data and either detrending or deseasonalization can dramatically reduce forecasting errors. Moreover‚ a combined detrending and deseasonalization is found to be the most effective data preprocessing approach.
    link
    BIB
    @article{zhang2005neural, series = {Decision {Support} {Systems} in the {Internet} {Age}}, title = {Neural network forecasting for seasonal and trend time series}, volume = {160}, issn = {0377-2217}, url = {http://www.sciencedirect.com/science/article/pii/ S0377221703005484}, doi = {10.1016/j.ejor.2003.08.037}, number = {2}, journal = {European Journal of Operational Research}, author = {Zhang, G. Peter and Qi, Min}, month = jan, year = {2005}, pages = {501--514} }
    RIS
    IEEE
    [1]
    G. P. Zhang and M. Qi, “Neural network forecasting for seasonal and trend time series,” European Journal of Operational Research, vol. 160, no. 2, pp. 501–514, Jan. 2005.
  42. ⊞ Forecasting seasonals and trends by exponentially weighted moving averages International Journal of Forecasting (January 2004)
    [1]
    C. C. Holt, “Forecasting seasonals and trends by exponentially weighted moving averages,” International Journal of Forecasting, vol. 20, no. 1, pp. 5–10, Jan. 2004.
    International Journal of Forecasting
    Abstract
    The paper provides a systematic development of the forecasting expressions for exponential weighted moving averages. Methods for series with no trend‚ or additive or multiplicative trend are examined. Similarly‚ the methods cover non-seasonal‚ and seasonal series with additive or multiplicative error structures. The paper is a reprinted version of the 1957 report to the Office of Naval Research (ONR 52) and is being published here to provide greater accessibility.
    link
    BIB
    @article{holt2004forecasting, title = {Forecasting seasonals and trends by exponentially weighted moving averages}, volume = {20}, issn = {0169-2070}, url = {http://www.sciencedirect.com/science/article/pii/ S0169207003001134}, doi = {10.1016/j.ijforecast.2003.09.015}, number = {1}, journal = {International Journal of Forecasting}, author = {Holt, Charles C.}, month = jan, year = {2004}, pages = {5--10} }
    RIS
    IEEE
    [1]
    C. C. Holt, “Forecasting seasonals and trends by exponentially weighted moving averages,” International Journal of Forecasting, vol. 20, no. 1, pp. 5–10, Jan. 2004.
  43. ⊞ Trend time series modeling and forecasting with neural networks (March 2003)
    [1]
    M. Qi and G. P. Zhang, “Trend time series modeling and forecasting with neural networks,” in 2003 IEEE International Conference on Computational Intelligence for Financial Engineering, 2003. Proceedings., 2003, pp. 331–337.
    00000
    Abstract
    Despite its great importance‚ there has been no general consensus on how to model the trends in time series data. Compared to traditional approaches‚ neural networks have shown some promise in time series forecasting. This paper investigates how to best model trend time series using neural networks. Four strategies (raw data‚ raw data with time index‚ detrending‚ and differencing) are used to model various simulated trend patterns (linear‚ nonlinear‚ deterministic‚ stochastic‚ and breaking trend). We find that with neural networks differencing often gives meritorious results regardless of the underlying DGPs. This finding is also confirmed by the real GNP series.
    BIB
    @inproceedings{qi2003trend, title = {Trend time series modeling and forecasting with neural networks}, doi = {10.1109/CIFER.2003.1196279}, booktitle = {2003 {IEEE} {International} {Conference} on {Computational} {Intelligence} for {Financial} {Engineering}, 2003. {Proceedings}.}, author = {Qi, Min and Zhang, G. P.}, month = mar, year = {2003}, note = {00000}, pages = {331--337} }
    RIS
    IEEE
    [1]
    M. Qi and G. P. Zhang, “Trend time series modeling and forecasting with neural networks,” in 2003 IEEE International Conference on Computational Intelligence for Financial Engineering, 2003. Proceedings., 2003, pp. 331–337.
  44. ⊞ A state space framework for automatic forecasting using exponential smoothing methods International Journal of Forecasting (July 2002)
    [1]
    R. J. Hyndman, A. B. Koehler, R. D. Snyder, and S. Grose, “A state space framework for automatic forecasting using exponential smoothing methods,” International Journal of Forecasting, vol. 18, no. 3, pp. 439–454, Jul. 2002.
    International Journal of Forecasting
    Abstract
    We provide a new approach to automatic forecasting based on an extended range of exponential smoothing methods. Each method in our taxonomy of exponential smoothing methods provides forecasts that are equivalent to forecasts from a state space model. This equivalence allows: (1) easy calculation of the likelihood‚ the AIC and other model selection criteria; (2) computation of prediction intervals for each method; and (3) random simulation from the underlying state space model. We demonstrate the methods by applying them to the data from the M-competition and the M3-competition. The method provides forecast accuracy comparable to the best methods in the competitions; it is particularly good for short forecast horizons with seasonal data.
    link
    BIB
    @article{hyndman2002state, title = {A state space framework for automatic forecasting using exponential smoothing methods}, volume = {18}, issn = {0169-2070}, url = {http://www.sciencedirect.com/science/article/pii/ S0169207001001108}, doi = {10.1016/S0169-2070(01)00110-8}, number = {3}, journal = {International Journal of Forecasting}, author = {Hyndman, Rob J and Koehler, Anne B and Snyder, Ralph D and Grose, Simone}, month = jul, year = {2002}, pages = {439--454} }
    RIS
    IEEE
    [1]
    R. J. Hyndman, A. B. Koehler, R. D. Snyder, and S. Grose, “A state space framework for automatic forecasting using exponential smoothing methods,” International Journal of Forecasting, vol. 18, no. 3, pp. 439–454, Jul. 2002.
  45. ⊞ SciPy: Open source scientific tools for Python (2001)
    [1]
    E. Jones, T. Oliphant, P. Peterson, and others, SciPy: Open source scientific tools for Python. 2001.
    00000
    link
    BIB
    @book{jones2001scipy:, title = {{SciPy}: {Open} source scientific tools for {Python}}, url = {http://www.scipy.org/}, author = {Jones, Eric and Oliphant, Travis and Peterson, Pearu and {others}}, year = {2001}, note = {00000} }
    RIS
    IEEE
    [1]
    E. Jones, T. Oliphant, P. Peterson, and others, SciPy: Open source scientific tools for Python. 2001.
  46. ⊞ Efficient BackProp (1998)
    [1]
    Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient BackProp,” in Neural Networks: Tricks of the Trade, Springer, Berlin, Heidelberg, 1998, pp. 9–50.
    DOI: 10.1007/3-540-49430-8_2
    Abstract
    The convergence of back-propagation learning is analyzed so as to explain common phenomenon observedb y practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposedin serious technical publications. This paper gives some of those tricks‚ ando.ers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most “classical” second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.
    link
    BIB
    @incollection{lecun1998efficient, series = {Lecture {Notes} in {Computer} {Science}}, title = {Efficient {BackProp}}, isbn = {978-3-540-65311-0 978-3-540-49430-0}, url = {https://link.springer.com/chapter/10.1007/3-540-49430-8_2}, language = {en}, booktitle = {Neural {Networks}: {Tricks} of the {Trade}}, publisher = {Springer, Berlin, Heidelberg}, author = {LeCun, Yann and Bottou, Leon and Orr, Genevieve B. and Müller, Klaus-Robert}, year = {1998}, doi = {10.1007/3-540-49430-8_2}, pages = {9--50} }
    RIS
    IEEE
    [1]
    Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient BackProp,” in Neural Networks: Tricks of the Trade, Springer, Berlin, Heidelberg, 1998, pp. 9–50.
  47. ⊞ New Capabilities and Methods of the X-12-ARIMA Seasonal-Adjustment Program Journal of Business & Economic Statistics (April 1998)
    [1]
    D. F. Findley, B. C. Monsell, W. R. Bell, M. C. Otto, and B.-C. Chen, “New Capabilities and Methods of the X-12-ARIMA Seasonal-Adjustment Program,” Journal of Business & Economic Statistics, vol. 16, no. 2, pp. 127–152, Apr. 1998.
    Journal of Business & Economic Statistics
    Abstract
    X-12-ARIMA is the Census Bureau’s new seasonal-adjustment program. It provides four types of enhancements to X-ll-ARIMA—(1) alternative seasonal‚ trading-day‚ and holiday effect adjustment capabilities that include adjustments for effects estimated with user-defined regressors; additional seasonal and trend filter options; and an alternative seasonal-trend-irregular decomposition; (2) new diagnostics of the quality and stability of the adjustments achieved under the options selected; (3) extensive time series modeling and model-selection capabilities for linear regression models with ARIMA errors‚ with optional robust estimation of coefficients; (4) a new user interface with features to facilitate batch processing large numbers of series.
    link
    BIB
    @article{findley1998new, title = {New {Capabilities} and {Methods} of the {X}-12-{ARIMA} {Seasonal}-{Adjustment} {Program}}, volume = {16}, issn = {0735-0015}, url = {http://www.tandfonline.com/doi/abs/10.1080/ 07350015.1998.10524743}, doi = {10.1080/07350015.1998.10524743}, number = {2}, journal = {Journal of Business \& Economic Statistics}, author = {Findley, David F. and Monsell, Brian C. and Bell, William R. and Otto, Mark C. and Chen, Bor-Chung}, month = apr, year = {1998}, pages = {127--152} }
    RIS
    IEEE
    [1]
    D. F. Findley, B. C. Monsell, W. R. Bell, M. C. Otto, and B.-C. Chen, “New Capabilities and Methods of the X-12-ARIMA Seasonal-Adjustment Program,” Journal of Business & Economic Statistics, vol. 16, no. 2, pp. 127–152, Apr. 1998.
  48. ⊞ Long Short-Term Memory Neural Computation (November 1997)
    [1]
    S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
    Neural Computation
    Abstract
    Learning to store information over extended time intervals by recurrent backpropagation takes a very long time‚ mostly because of insufficient‚ decaying error backflow. We briefly review Hochreiter’s (1991) analysis of this problem‚ then address it by introducing a novel‚ efficient‚ gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm‚ LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local‚ distributed‚ real-valued‚ and noisy pattern representations. In comparisons with real-time recurrent learning‚ back propagation through time‚ recurrent cascade correlation‚ Elman nets‚ and neural sequence chunking‚ LSTM leads to many more successful runs‚ and learns much faster. LSTM also solves complex‚ artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.
    link
    BIB
    @article{hochreiter1997long, title = {Long {Short}-{Term} {Memory}}, volume = {9}, issn = {0899-7667}, url = {https://doi.org/10.1162/neco.1997.9.8.1735}, doi = {10.1162/neco.1997.9.8.1735}, number = {8}, journal = {Neural Computation}, author = {Hochreiter, Sepp and Schmidhuber, Jürgen}, month = nov, year = {1997}, pages = {1735--1780} }
    RIS
    IEEE
    [1]
    S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
  49. ⊞ Multitask Learning Machine Learning (July 1997)
    [1]
    R. Caruana, “Multitask Learning,” Machine Learning, vol. 28, no. 1, pp. 41–75, Jul. 1997.
    02964
    Machine Learning
    Abstract
    Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better. This paper reviews prior work on MTL‚ presents new evidence that MTL in backprop nets discovers task relatedness without the need of supervisory signals‚ and presents new results for MTL with k-nearest neighbor and kernel regression. In this paper we demonstrate multitask learning in three domains. We explain how multitask learning works‚ and show that there are many opportunities for multitask learning in real domains. We present an algorithm and results for multitask learning with case-based methods like k-nearest neighbor and kernel regression‚ and sketch an algorithm for multitask learning in decision trees. Because multitask learning works‚ can be applied to many different kinds of domains‚ and can be used with different learning algorithms‚ we conjecture there will be many opportunities for its use on real-world problems.
    link
    BIB
    @article{caruana1997multitask, title = {Multitask {Learning}}, volume = {28}, issn = {0885-6125, 1573-0565}, url = {https://link.springer.com/article/10.1023/A:1007379606734}, doi = {10.1023/A:1007379606734}, language = {en}, number = {1}, journal = {Machine Learning}, author = {Caruana, Rich}, month = jul, year = {1997}, note = {02964}, pages = {41--75} }
    RIS
    IEEE
    [1]
    R. Caruana, “Multitask Learning,” Machine Learning, vol. 28, no. 1, pp. 41–75, Jul. 1997.
  50. ⊞ Declarative bias in equation discovery (1997)
    [1]
    L. Todorovski and S. Dzeroski, “Declarative bias in equation discovery,” in ICML, 1997, pp. 376–384.
    00150
    BIB
    @inproceedings{todorovski1997declarative, title = {Declarative bias in equation discovery}, booktitle = {{ICML}}, author = {Todorovski, Ljupco and Dzeroski, Saso}, year = {1997}, note = {00150}, pages = {376--384} }
    RIS
    IEEE
    [1]
    L. Todorovski and S. Dzeroski, “Declarative bias in equation discovery,” in ICML, 1997, pp. 376–384.
  51. ⊞ Design and regularization of neural networks: the optimal use of a validation set (September 1996)
    [1]
    J. Larsen, L. K. Hansen, C. Svarer, and M. Ohlsson, “Design and regularization of neural networks: the optimal use of a validation set,” in Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop, 1996, pp. 62–71.
    Abstract
    We derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularisation parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based optimal brain damage/surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate the viability of the methods
    BIB
    @inproceedings{larsen1996design, title = {Design and regularization of neural networks: the optimal use of a validation set}, doi = {10.1109/NNSP.1996.548336}, booktitle = {Neural {Networks} for {Signal} {Processing} {VI}. {Proceedings} of the 1996 {IEEE} {Signal} {Processing} {Society} {Workshop}}, author = {Larsen, J. and Hansen, L. K. and Svarer, C. and Ohlsson, M.}, month = sep, year = {1996}, pages = {62--71} }
    RIS
    IEEE
    [1]
    J. Larsen, L. K. Hansen, C. Svarer, and M. Ohlsson, “Design and regularization of neural networks: the optimal use of a validation set,” in Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop, 1996, pp. 62–71.
  52. ⊞ Designing a neural network for forecasting financial and economic time series Neurocomputing (April 1996)
    [1]
    I. Kaastra and M. Boyd, “Designing a neural network for forecasting financial and economic time series,” Neurocomputing, vol. 10, no. 3, pp. 215–236, Apr. 1996.
    00000
    Neurocomputing
    Abstract
    Artificial neural networks are universal and highly flexible function approximators first used in the fields of cognitive science and engineering. In recent years‚ neural network applications in finance for such tasks as pattern recognition‚ classification‚ and time series forecasting have dramatically increased. However‚ the large number of parameters that must be selected to develop a neural network forecasting model have meant that the design process still involves much trial and error. The objective of this paper is to provide a practical introductory guide in the design of a neural network for forecasting economic time series data. An eight-step procedure to design a neural network forecasting model is explained including a discussion of tradeoffs in parameter selection‚ some common pitfalls‚ and points of disagreement among practitioners.
    link
    BIB
    @article{kaastra1996designing, series = {Financial {Applications}, {Part} {II}}, title = {Designing a neural network for forecasting financial and economic time series}, volume = {10}, issn = {0925-2312}, url = {http://www.sciencedirect.com/science/article/pii/ 0925231295000399}, doi = {10.1016/0925-2312(95)00039-9}, number = {3}, journal = {Neurocomputing}, author = {Kaastra, Iebeling and Boyd, Milton}, month = apr, year = {1996}, note = {00000}, pages = {215--236} }
    RIS
    IEEE
    [1]
    I. Kaastra and M. Boyd, “Designing a neural network for forecasting financial and economic time series,” Neurocomputing, vol. 10, no. 3, pp. 215–236, Apr. 1996.
  53. ⊞ Particle swarm optimization (November 1995)
    [1]
    J. Kennedy and R. Eberhart, “Particle swarm optimization,” in , IEEE International Conference on Neural Networks, 1995. Proceedings, 1995, vol. 4, pp. 1942–1948 vol.4.
    00000
    Abstract
    A concept for the optimization of nonlinear functions using particle swarm methodology is introduced. The evolution of several paradigms is outlined‚ and an implementation of one of the paradigms is discussed. Benchmark testing of the paradigm is described‚ and applications‚ including nonlinear function optimization and neural network training‚ are proposed. The relationships between particle swarm optimization and both artificial life and genetic algorithms are described
    BIB
    @inproceedings{kennedy1995particle, title = {Particle swarm optimization}, volume = {4}, doi = {10.1109/ICNN.1995.488968}, booktitle = {, {IEEE} {International} {Conference} on {Neural} {Networks}, 1995. {Proceedings}}, author = {Kennedy, J. and Eberhart, R.}, month = nov, year = {1995}, note = {00000}, pages = {1942--1948 vol.4} }
    RIS
    IEEE
    [1]
    J. Kennedy and R. Eberhart, “Particle swarm optimization,” in , IEEE International Conference on Neural Networks, 1995. Proceedings, 1995, vol. 4, pp. 1942–1948 vol.4.
  54. ⊞ Learning long-term dependencies with gradient descent is difficult IEEE Transactions on Neural Networks (March 1994)
    [1]
    Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157–166, Mar. 1994.
    IEEE Transactions on Neural Networks
    Abstract
    Recurrent neural networks can be used to map input sequences to output sequences‚ such as for recognition‚ production or prediction problems. However‚ practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem‚ alternatives to standard gradient descent are considered
    BIB
    @article{bengio1994learning, title = {Learning long-term dependencies with gradient descent is difficult}, volume = {5}, issn = {1045-9227}, doi = {10.1109/72.279181}, number = {2}, journal = {IEEE Transactions on Neural Networks}, author = {Bengio, Y. and Simard, P. and Frasconi, P.}, month = mar, year = {1994}, pages = {157--166} }
    RIS
    IEEE
    [1]
    Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157–166, Mar. 1994.
  55. ⊞ Chapter 49 Arch models (January 1994)
    [1]
    T. Bollerslev, R. F. Engle, and D. B. Nelson, “Chapter 49 Arch models,” in Handbook of Econometrics, vol. 4, Elsevier, 1994, pp. 2959–3038.
    DOI: 10.1016/S1573-4412(05)80018-2
    Abstract
    This chapter evaluates the most important theoretical developments in ARCH type modeling of time-varying conditional variances. The coverage include the specification of univariate parametric ARCH models‚ general inference procedures‚ conditions for stationarity and ergodicity‚ continuous time methods‚ aggregation and forecasting of ARCH models‚ multivariate conditional covariance formulations‚ and the use of model selection criteria in an ARCH context. Additionally‚ the chapter contains a discussion of the empirical regularities pertaining to the temporal variation in financial market volatility. Motivated in part by recent results on optimal filtering‚ a new conditional variance model for better characterizing stock return volatility is also presented.
    link
    BIB
    @incollection{bollerslev1994chapter, title = {Chapter 49 {Arch} models}, volume = {4}, url = {http://www.sciencedirect.com/science/article/pii/ S1573441205800182}, booktitle = {Handbook of {Econometrics}}, publisher = {Elsevier}, author = {Bollerslev, Tim and Engle, Robert F. and Nelson, Daniel B.}, month = jan, year = {1994}, doi = {10.1016/S1573-4412(05)80018-2}, pages = {2959--3038} }
    RIS
    IEEE
    [1]
    T. Bollerslev, R. F. Engle, and D. B. Nelson, “Chapter 49 Arch models,” in Handbook of Econometrics, vol. 4, Elsevier, 1994, pp. 2959–3038.
  56. ⊞ Neural Net Architectures for Temporal Sequence Processing (March 1993)
    [1]
    M. Mozer, “Neural Net Architectures for Temporal Sequence Processing,” Mar. 1993.
    00381
    Abstract
    I present a general taxonomy of neural net architectures for processing time-varying patterns. This taxonomy subsumes many existing architectures in the literature‚ and points to several promising architectures that have yet to be examined. Any architecture that processes timevarying patterns requires two conceptually distinct components: a short-term memory that holds on to relevant past events and an associator that uses the short-term memory to classify or predict. My taxonomy is based on a characterization of short-term memory models along the dimensions of form‚ content‚ and adaptability. Experiments on predicting future values of a financial time series (US dollar–Swiss franc exchange rates) are presented using several alternative memory models. The results of these experiments serve as a baseline against which more sophisticated architectures can be compared. Neural networks have proven to be a promising alternative to traditional techniques for nonlinear temporal pre...
    BIB
    @article{mozer1993neural, title = {Neural {Net} {Architectures} for {Temporal} {Sequence} {Processing}}, author = {Mozer, Michael}, month = mar, year = {1993}, note = {00381} }
    RIS
    IEEE
    [1]
    M. Mozer, “Neural Net Architectures for Temporal Sequence Processing,” Mar. 1993.
  57. ⊞ Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? Journal of Econometrics (October 1992)
    [1]
    D. Kwiatkowski, P. C. B. Phillips, P. Schmidt, and Y. Shin, “Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root?,” Journal of Econometrics, vol. 54, no. 1, pp. 159–178, Oct. 1992.
    10948
    Journal of Econometrics
    Abstract
    We propose a test of the null hypothesis that an observable series is stationary around a deterministic trend. The series is expressed as the sum of deterministic trend‚ random walk‚ and stationary error‚ and the test is the LM test of the hypothesis that the random walk has zero variance. The asymptotic distribution of the statistic is derived under the null and under the alternative that the series is difference-stationary. Finite sample size and power are considered in a Monte Carlo experiment. The test is applied to the Nelson-Plosser data‚ and for many of these series the hypothesis of trend stationarity cannot be rejected.
    link
    BIB
    @article{kwiatkowski1992testing, title = {Testing the null hypothesis of stationarity against the alternative of a unit root: {How} sure are we that economic time series have a unit root?}, volume = {54}, issn = {0304-4076}, url = {http://www.sciencedirect.com/science/article/pii/ 030440769290104Y}, doi = {10.1016/0304-4076(92)90104-Y}, number = {1}, journal = {Journal of Econometrics}, author = {Kwiatkowski, Denis and Phillips, Peter C. B. and Schmidt, Peter and Shin, Yongcheol}, month = oct, year = {1992}, note = {10948}, pages = {159--178} }
    RIS
    IEEE
    [1]
    D. Kwiatkowski, P. C. B. Phillips, P. Schmidt, and Y. Shin, “Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root?,” Journal of Econometrics, vol. 54, no. 1, pp. 159–178, Oct. 1992.
  58. ⊞ Backpropagation Applied to Handwritten Zip Code Recognition Neural Computation (December 1989)
    [1]
    Y. LeCun et al., “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, Dec. 1989.
    Neural Computation
    Abstract
    The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation‚ going from the normalized image of the character to the final classification.
    link
    BIB
    @article{lecun1989backpropagation, title = {Backpropagation {Applied} to {Handwritten} {Zip} {Code} {Recognition}}, volume = {1}, issn = {0899-7667}, url = {https://doi.org/10.1162/neco.1989.1.4.541}, doi = {10.1162/neco.1989.1.4.541}, number = {4}, journal = {Neural Computation}, author = {LeCun, Y. and Boser, B. and Denker, J. S. and Henderson, D. and Howard, R. E. and Hubbard, W. and Jackel, L. D.}, month = dec, year = {1989}, pages = {541--551} }
    RIS
    IEEE
    [1]
    Y. LeCun et al., “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, Dec. 1989.
  59. ⊞ Multilayer feedforward networks are universal approximators Neural Networks (January 1989)
    [1]
    K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, Jan. 1989.
    Neural Networks
    Abstract
    This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy‚ provided sufficiently many hidden units are available. In this sense‚ multilayer feedforward networks are a class of universal approximators.
    link
    BIB
    @article{hornik1989multilayer, title = {Multilayer feedforward networks are universal approximators}, volume = {2}, issn = {0893-6080}, url = {http://www.sciencedirect.com/science/article/pii/ 0893608089900208}, doi = {10.1016/0893-6080(89)90020-8}, number = {5}, journal = {Neural Networks}, author = {Hornik, Kurt and Stinchcombe, Maxwell and White, Halbert}, month = jan, year = {1989}, pages = {359--366} }
    RIS
    IEEE
    [1]
    K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, Jan. 1989.
  60. ⊞ Testing for a Unit Root in Time Series Regression Biometrika (1988)
    [1]
    P. C. B. Phillips and P. Perron, “Testing for a Unit Root in Time Series Regression,” Biometrika, vol. 75, no. 2, pp. 335–346, 1988.
    Biometrika
    Biometrika
    Abstract
    This paper proposes new tests for detecting the presence of a unit root in quite general time series models. Our approach is nonparametric with respect to nuisance parameters and thereby allows for a very wide class of weakly dependent and possibly heterogeneously distributed data. The tests accommodate models with a fitted drift and a time trend so that they may be used to discriminate between unit root nonstationarity and stationarity about a deterministic trend. The limiting distributions of the statistics are obtained under both the unit root null and a sequence of local alternatives. The latter noncentral distribution theory yields local asymptotic power functions for the tests and facilitates comparisons with alternative procedures due to Dickey & Fuller. Simulations are reported on the performance of the new tests in finite samples.
    link
    BIB
    @article{phillips1988testing, title = {Testing for a {Unit} {Root} in {Time} {Series} {Regression}}, volume = {75}, issn = {0006-3444}, url = {http://www.jstor.org/stable/2336182}, doi = {10.2307/2336182}, number = {2}, journal = {Biometrika}, author = {Phillips, Peter C. B. and Perron, Pierre}, year = {1988}, pages = {335--346} }
    RIS
    IEEE
    [1]
    P. C. B. Phillips and P. Perron, “Testing for a Unit Root in Time Series Regression,” Biometrika, vol. 75, no. 2, pp. 335–346, 1988.
  61. ⊞ Generalization of backpropagation with application to a recurrent gas market model Neural Networks (January 1988)
    [1]
    P. J. Werbos, “Generalization of backpropagation with application to a recurrent gas market model,” Neural Networks, vol. 1, no. 4, pp. 339–356, Jan. 1988.
    00712
    Neural Networks
    Abstract
    Backpropagation is often viewed as a method for adapting artificial neural networks to classify patterns. Based on parts of the book by Rumelhart and colleagues‚ many authors equate backpropagation with the generalized delta rule applied to fully-connected feedforward networks. This paper will summarize a more general formulation of backpropagation‚ developed in 1974‚ which does more justice to the roots of the method in numerical analysis and statistics‚ and also does more justice to creative approaches expressed by neural modelers in the past year or two. It will discuss applications of backpropagation to forecasting over time (where errors have been halved by using methods other than least squares)‚ to optimization‚ to sensitivity analysis‚ and to brain research. This paper will go on to derive a generalization of backpropagation to recurrent systems (which input their own output)‚ such as hybrids of perceptron-style networks and Grossberg/Hopfield networks. Unlike the proposal of Rumelhart‚ Hinton‚ and Williams‚ this generalization does not require the storage of intermediate iterations to deal with continuous recurrence. This generalization was applied in 1981 to a model of natural gas markets‚ where it located sources of forecast uncertainty related to the use of least squares to estimate the model parameters in the first place.
    link
    BIB
    @article{werbos1988generalization, title = {Generalization of backpropagation with application to a recurrent gas market model}, volume = {1}, issn = {0893-6080}, url = {http://www.sciencedirect.com/science/article/pii/ 089360808890007X}, doi = {10.1016/0893-6080(88)90007-X}, number = {4}, journal = {Neural Networks}, author = {Werbos, Paul J.}, month = jan, year = {1988}, note = {00712}, pages = {339--356} }
    RIS
    IEEE
    [1]
    P. J. Werbos, “Generalization of backpropagation with application to a recurrent gas market model,” Neural Networks, vol. 1, no. 4, pp. 339–356, Jan. 1988.
  62. ⊞ The Utility Driven Dynamic Error Propagation Network (1987)
    [1]
    A. J. Robinson and F. Fallside, “The Utility Driven Dynamic Error Propagation Network,” Engineering Department, Cambridge University, Cambridge, UK, CUED/F-INFENG/TR.1, 1987.
    BIB
    @techreport{robinson1987utility, address = {Cambridge, UK}, title = {The {Utility} {Driven} {Dynamic} {Error} {Propagation} {Network}}, number = {CUED/F-INFENG/TR.1}, institution = {Engineering Department, Cambridge University}, author = {Robinson, A. J. and Fallside, Frank}, year = {1987} }
    RIS
    IEEE
    [1]
    A. J. Robinson and F. Fallside, “The Utility Driven Dynamic Error Propagation Network,” Engineering Department, Cambridge University, Cambridge, UK, CUED/F-INFENG/TR.1, 1987.
  63. ⊞ Forecasting the volatility of currency exchange rates International Journal of Forecasting (January 1987)
    [1]
    S. J. Taylor, “Forecasting the volatility of currency exchange rates,” International Journal of Forecasting, vol. 3, no. 1, pp. 159–170, Jan. 1987.
    International Journal of Forecasting
    Abstract
    Currency volatility is defined to be the standard deviation of day-to-day changes in the logarithm of the exchange rate. After a discussion of statistical models for exchange rates‚ the paper describes methods for choosing and assessing volatility forecasts using open‚ high‚ low and close prices. Results for DM/ futures prices at the IMM in Chicago from 1977 to 1983 show high and low prices are valuable when seeking accurate volatility forecasts. The best forecasts are a weighted average of present and past high‚ low and close prices‚ with adjustments for weekend and holiday effects. The forecasts can be used to value currency options.
    link
    BIB
    @article{taylor1987forecasting, series = {Special {Issue} on {Exchange} {Rate} {Forecasting}}, title = {Forecasting the volatility of currency exchange rates}, volume = {3}, issn = {0169-2070}, url = {http://www.sciencedirect.com/science/article/pii/ 0169207087900859}, doi = {10.1016/0169-2070(87)90085-9}, number = {1}, journal = {International Journal of Forecasting}, author = {Taylor, Stephen J.}, month = jan, year = {1987}, pages = {159--170} }
    RIS
    IEEE
    [1]
    S. J. Taylor, “Forecasting the volatility of currency exchange rates,” International Journal of Forecasting, vol. 3, no. 1, pp. 159–170, Jan. 1987.
  64. ⊞ Learning Internal Representations by Error Propagation (September 1985)
    [1]
    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Internal Representations by Error Propagation,” CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE, CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE, ICS-8506, Sep. 1985.
    Abstract
    This paper presents a generalization of the perception learning procedure for learning the correct sets of connections for arbitrary networks. The rule‚ falled the generalized delta rule‚ is a simple scheme for implementing a gradient descent method for finding weights that minimize the sum squared error of the sytem’s performance. The major theoretical contribution of the work is the procedure called error propagation‚ whereby the gradient can be determined by individual units of the network based only on locally available information. The major empirical contribution of the work is to show that the problem of local minima not serious in this application of gradient descent. Keywords: Learning; networks; Perceptrons; Adaptive systems; Learning machines; and Back propagation.
    link
    BIB
    @techreport{rumelhart1985learning, title = {Learning {Internal} {Representations} by {Error} {Propagation}}, url = {http://www.dtic.mil/docs/citations/ADA164453}, language = {en}, number = {ICS-8506}, institution = {CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE, CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE}, author = {Rumelhart, David E. and Hinton, Geoffrey E. and Williams, Ronald J.}, month = sep, year = {1985} }
    RIS
    IEEE
    [1]
    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Internal Representations by Error Propagation,” CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE, CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE, ICS-8506, Sep. 1985.
  65. ⊞ Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation Econometrica (1982)
    [1]
    R. F. Engle, “Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation,” Econometrica, vol. 50, no. 4, pp. 987–1007, 1982.
    23957
    Econometrica
    Abstract
    Traditional econometric models assume a constant one-period forecast variance. To generalize this implausible assumption‚ a new class of stochastic processes called autoregressive conditional heteroscedastic (ARCH) processes are introduced in this paper. These are mean zero‚ serially uncorrelated processes with nonconstant variances conditional on the past‚ but constant unconditional variances. For such processes‚ the recent past gives information about the one-period forecast variance. A regression model is then introduced with disturbances following an ARCH process. Maximum likelihood estimators are described and a simple scoring iteration formulated. Ordinary least squares maintains its optimality properties in this set-up‚ but maximum likelihood is more efficient. The relative efficiency is calculated and can be infinite. To test whether the disturbances follow an ARCH process‚ the Lagrange multiplier procedure is employed. The test is based simply on the autocorrelation of the squared OLS residuals. This model is used to estimate the means and variances of inflation in the U.K. The ARCH effect is found to be significant and the estimated variances increase substantially during the chaotic seventies.
    link
    BIB
    @article{engle1982autoregressive, title = {Autoregressive {Conditional} {Heteroscedasticity} with {Estimates} of the {Variance} of {United} {Kingdom} {Inflation}}, volume = {50}, issn = {0012-9682}, url = {http://www.jstor.org/stable/1912773}, doi = {10.2307/1912773}, number = {4}, journal = {Econometrica}, author = {Engle, Robert F.}, year = {1982}, note = {23957}, pages = {987--1007} }
    RIS
    IEEE
    [1]
    R. F. Engle, “Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation,” Econometrica, vol. 50, no. 4, pp. 987–1007, 1982.
  66. ⊞ Distribution of the Estimators for Autoregressive Time Series With a Unit Root Journal of the American Statistical Association (1979)
    [1]
    D. A. Dickey and W. A. Fuller, “Distribution of the Estimators for Autoregressive Time Series With a Unit Root,” Journal of the American Statistical Association, vol. 74, no. 366, pp. 427–431, 1979.
    Journal of the American Statistical Association
    Abstract
    Let n observations Y_\textrm1
    link
    BIB
    @article{dickey1979distribution, title = {Distribution of the {Estimators} for {Autoregressive} {Time} {Series} {With} a {Unit} {Root}}, volume = {74}, issn = {0162-1459}, url = {http://www.jstor.org/stable/2286348}, doi = {10.2307/2286348}, $$_{\textrm{2}}$$_{\textrm{n}}$$_{\textrm{t}}$$_{\textrm{t - 1}}$$_{\textrm{t}}$$_{\textrm{0}}$\$ is a fixed constant and \{\vphantom{\}}e$_{\textrm{t}}$$_{\textrm{t = 1}}$$^{\textrm{n}}$$^{\textrm{2}}$\$. Properties of the regression estimator of ρ are obtained under the assumption that ρ = ± 1. Representations for the limit distributions of the estimator of ρ and of the regression t test are derived. The estimator of ρ and the regression t test furnish methods of testing the hypothesis that ρ = 1.}, number = {366}, journal = {Journal of the American Statistical Association}, author = {Dickey, David A. and Fuller, Wayne A.}, year = {1979}, pages = {427--431} }
    RIS
    IEEE
    [1]
    D. A. Dickey and W. A. Fuller, “Distribution of the Estimators for Autoregressive Time Series With a Unit Root,” Journal of the American Statistical Association, vol. 74, no. 366, pp. 427–431, 1979.
  67. ⊞ Beyond regression: new tools for prediction and analysis in the behavioral sciences (1975)
    [1]
    P. J. Werbos, “Beyond regression: new tools for prediction and analysis in the behavioral sciences,” PhD Thesis, 1975.
    00014
    BIB
    @phdthesis{werbos1975beyond, type = {{PhD} {Thesis}}, title = {Beyond regression: new tools for prediction and analysis in the behavioral sciences}, language = {English}, author = {Werbos, Paul J}, year = {1975}, note = {00014} }
    RIS
    IEEE
    [1]
    P. J. Werbos, “Beyond regression: new tools for prediction and analysis in the behavioral sciences,” PhD Thesis, 1975.
  68. ⊞ Adaptive switching circuits (1960)
    [1]
    B. Widrow and M. E. Hoff, “Adaptive switching circuits,” STANFORD UNIV CA STANFORD ELECTRONICS LABS, 1960.
    05223
    BIB
    @techreport{widrow1960adaptive, title = {Adaptive switching circuits}, institution = {STANFORD UNIV CA STANFORD ELECTRONICS LABS}, author = {Widrow, Bernard and Hoff, Marcian E}, year = {1960}, note = {05223} }
    RIS
    IEEE
    [1]
    B. Widrow and M. E. Hoff, “Adaptive switching circuits,” STANFORD UNIV CA STANFORD ELECTRONICS LABS, 1960.
  69. ⊞ Forecasting Sales by Exponentially Weighted Moving Averages Management Science (April 1960)
    [1]
    P. R. Winters, “Forecasting Sales by Exponentially Weighted Moving Averages,” Management Science, vol. 6, no. 3, pp. 324–342, Apr. 1960.
    01602
    Management Science
    Abstract
    The growing use of computers for mechanized inventory control and production planning has brought with it the need for explicit forecasts of sales and usage for individual products and materials. These forecasts must be made on a routine basis for thousands of products‚ so that they must be made quickly‚ and‚ both in terms of computing time and information storage‚ cheaply; they should be responsive to changing conditions. The paper presents a method of forecasting sales which has these desirable characteristics‚ and which in terms of ability to forecast compares favorably with other‚ more traditional methods. Several models of the exponential forecasting system are presented‚ along with several examples of application.
    link
    BIB
    @article{winters1960forecasting, title = {Forecasting {Sales} by {Exponentially} {Weighted} {Moving} {Averages}}, volume = {6}, issn = {0025-1909}, url = {https://pubsonline.informs.org/doi/abs/10.1287/ mnsc.6.3.324}, doi = {10.1287/mnsc.6.3.324}, number = {3}, journal = {Management Science}, author = {Winters, Peter R.}, month = apr, year = {1960}, note = {01602}, pages = {324--342} }
    RIS
    IEEE
    [1]
    P. R. Winters, “Forecasting Sales by Exponentially Weighted Moving Averages,” Management Science, vol. 6, no. 3, pp. 324–342, Apr. 1960.
  70. ⊞ The perceptron: A probabilistic model for information storage and organization in the brain Psychological Review (1958)
    [1]
    F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, no. 6, pp. 386–408, 1958.
    07783
    Psychological Review
    Abstract
    To answer the questions of how information about the physical world is sensed‚ in what form is information remembered‚ and how does information retained in memory influence recognition and behavior‚ a theory is developed for a hypothetical nervous system called a perceptron. The theory serves as a bridge between biophysics and psychology. It is possible to predict learning curves from neurological variables and vice versa. The quantitative statistical approach is fruitful in the understanding of the organization of cognitive systems. 18 references. (PsycINFO Database Record (c) 2016 APA‚ all rights reserved)
    BIB
    @article{rosenblatt1958perceptron:, title = {The perceptron: {A} probabilistic model for information storage and organization in the brain}, volume = {65}, issn = {1939-1471(Electronic),0033-295X(Print)}, doi = {10.1037/h0042519}, number = {6}, journal = {Psychological Review}, author = {Rosenblatt, F.}, year = {1958}, note = {07783}, pages = {386--408} }
    RIS
    IEEE
    [1]
    F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, no. 6, pp. 386–408, 1958.
  71. ⊞ Individual Comparisons by Ranking Methods Biometrics Bulletin (1945)
    [1]
    F. Wilcoxon, “Individual Comparisons by Ranking Methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945.
    00000
    Biometrics Bulletin
    link
    BIB
    @article{wilcoxon1945individual, title = {Individual {Comparisons} by {Ranking} {Methods}}, volume = {1}, issn = {0099-4987}, url = {http://www.jstor.org/stable/3001968}, doi = {10.2307/3001968}, number = {6}, journal = {Biometrics Bulletin}, author = {Wilcoxon, Frank}, year = {1945}, note = {00000}, pages = {80--83} }
    RIS
    IEEE
    [1]
    F. Wilcoxon, “Individual Comparisons by Ranking Methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945.
  72. ⊞ A logical calculus of the ideas immanent in nervous activity The bulletin of mathematical biophysics (December 1943)
    [1]
    W. S. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The bulletin of mathematical biophysics, vol. 5, no. 4, pp. 115–133, Dec. 1943.
    15658
    The bulletin of mathematical biophysics
    Abstract
    Because of the “all-or-none” character of nervous activity‚ neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms‚ with the addition of more complicated logical means for nets containing circles; and that for any logical expression satisfying certain conditions‚ one can find a net behaving in the fashion it describes. It is shown that many particular choices among possible neurophysiological assumptions are equivalent‚ in the sense that for every net behaving under one assumption‚ there exists another net which behaves under the other and gives the same results‚ although perhaps not in the same time. Various applications of the calculus are discussed.
    link
    BIB
    @article{mcculloch1943logical, title = {A logical calculus of the ideas immanent in nervous activity}, volume = {5}, issn = {0007-4985, 1522-9602}, url = {https://link.springer.com/article/10.1007/BF02478259}, doi = {10.1007/BF02478259}, language = {en}, number = {4}, journal = {The bulletin of mathematical biophysics}, author = {McCulloch, Warren S. and Pitts, Walter}, month = dec, year = {1943}, note = {15658}, pages = {115--133} }
    RIS
    IEEE
    [1]
    W. S. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The bulletin of mathematical biophysics, vol. 5, no. 4, pp. 115–133, Dec. 1943.
  73. ⊞ 6.4.4.3.1. Seasonal Subseries Plot
    [1]
    6.4.4.3.1. Seasonal Subseries Plot. .
    00000
    link
    BIB
    @book{noauthornodate6.4.4.3.1., title = {6.4.4.3.1. {Seasonal} {Subseries} {Plot}}, url = {https://www.itl.nist.gov/div898/handbook/pmc/section4/ pmc4431.htm}, note = {00000} }
    RIS
    IEEE
    [1]
    6.4.4.3.1. Seasonal Subseries Plot. .
  74. ⊞ Practical Bayesian Optimization of Machine Learning Algorithms
    [1]
    J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” p. 9.
    Abstract
    The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately‚ this tuning is often a “black art” requiring expert experience‚ rules of thumb‚ or sometimes bruteforce search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work‚ we consider this problem through the framework of Bayesian optimization‚ in which a learning algorithm’s generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP‚ such as the type of kernel and the treatment of its hyperparameters‚ can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation‚ structured SVMs and convolutional neural networks.
    BIB
    @article{snoeknodatepractical, title = {Practical {Bayesian} {Optimization} of {Machine} {Learning} {Algorithms}}, language = {en}, author = {Snoek, Jasper and Larochelle, Hugo and Adams, Ryan P}, pages = {9} }
    RIS
    IEEE
    [1]
    J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” p. 9.
  75. ⊞ Random Search for Hyper-Parameter Optimization
    [1]
    J. Bergstra, Y. Bengio, J. BERGSTRA, and Y. BENGIO, “Random Search for Hyper-Parameter Optimization,” p. 25.
    01474
    Abstract
    Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search‚ we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget‚ random search finds better models by effectively searching a larger‚ less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search‚ purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets‚ and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter‚ but that different hyper-parameters are important on different data sets. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. Our analysis casts some light on why recent “High Throughput” methods achieve surprising success—they appear to search through a large number of hyper-parameters because most hyper-parameters do not matter much. We anticipate that growing interest in large hierarchical models will place an increasing burden on techniques for hyper-parameter optimization; this work shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms.
    BIB
    @article{bergstranodaterandom, title = {Random {Search} for {Hyper}-{Parameter} {Optimization}}, language = {en}, author = {Bergstra, James and Bengio, Yoshua and BERGSTRA, JAMES and BENGIO, YOSHUA}, note = {01474}, pages = {25} }
    RIS
    IEEE
    [1]
    J. Bergstra, Y. Bengio, J. BERGSTRA, and Y. BENGIO, “Random Search for Hyper-Parameter Optimization,” p. 25.
  76. ⊞ Understanding the diffculty of training deep feedforward neural networks
    [1]
    X. Glorot and Y. Bengio, “Understanding the diffculty of training deep feedforward neural networks,” p. 8.
    00000
    Abstract
    Whereas before 2006 it appears that deep multilayer neural networks were not successfully trained‚ since then several algorithms have been shown to successfully train them‚ with experimental results showing the superiority of deeper vs less deep architectures. All these experimental results were obtained with new initialization or training mechanisms. Our objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks‚ to better understand these recent relative successes and help design better algorithms in the future. We first observe the influence of the non-linear activations functions. We find that the logistic sigmoid activation is unsuited for deep networks with random initialization because of its mean value‚ which can drive especially the top hidden layer into saturation. Surprisingly‚ we find that saturated units can move out of saturation by themselves‚ albeit slowly‚ and explaining the plateaus sometimes seen when training neural networks. We find that a new non-linearity that saturates less can often be beneficial. Finally‚ we study how activations and gradients vary across layers and during training‚ with the idea that training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1. Based on these considerations‚ we propose a new initialization scheme that brings substantially faster convergence.
    BIB
    @article{glorotnodateunderstanding, title = {Understanding the diffculty of training deep feedforward neural networks}, language = {en}, author = {Glorot, Xavier and Bengio, Yoshua}, note = {00000}, pages = {8} }
    RIS
    IEEE
    [1]
    X. Glorot and Y. Bengio, “Understanding the diffculty of training deep feedforward neural networks,” p. 8.
  77. ⊞ Algorithms for Hyper-Parameter Optimization
    [1]
    J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for Hyper-Parameter Optimization,” p. 9.
    00566
    Abstract
    Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel approaches to feature learning. Traditionally‚ hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently‚ computer clusters and GPU processors make it possible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neural networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the expected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets‚ but we show it is unreliable for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P (y\textbackslashtextbarx) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.
    BIB
    @article{bergstranodatealgorithms, title = {Algorithms for {Hyper}-{Parameter} {Optimization}}, language = {en}, author = {Bergstra, James S and Bardenet, Rémi and Bengio, Yoshua and Kégl, Balázs}, note = {00566}, pages = {9} }
    RIS
    IEEE
    [1]
    J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for Hyper-Parameter Optimization,” p. 9.
A zot_bib_web bibliography.