Deep Learning – Sequence Data
Automobile parts reliability prediction based on claim data: The comparison of predictive effects with Deep Learning
Objective
Reliability prediction, Automobile claim data to investigate various reliability prediction models using parametric method, time-series analysis, machine learning, and deep learning-based methods with a dataset of the reliability time series
Data
In this study, a warranty data called field claim was used to record the purchased car and its related causes of failure, parts, days of use, mileage, etc. for a quality repair. These data are key data that provide feedback on the life expectancy of the product and contain all failures within the warranty period. Among these field claim data, data from X-model automobile 06MY–14MY claims were extracted in the US market, which these data were made into virtual data by adding a little noise due to security issues and all were masked. Claim data is recorded and stored in the database when A/S is received, and 305,965 claims are extracted for study and recorded.
Related Work
The RNN can learn how to forget or store information at each point in time, so it shows good performance especially in time-series data processing. However, when the time point to be predicted increases, a gradient vanishing problem could occur and the efficiency of the learning process decreases. RNN can learn how to forget or store information at each point in time, so it shows good performance especially in timeseries data processing. However, when the time point to be predicted increases, a gradient vanishing problem occurs, resulting in poor learning efficiency. To solve this problem, Hochreiter et al (1997) proposed long short term memory (LSTM). Unlike RNNs, LSTMs are designed to handle data with long periods of time and are used in various fields such as classification and time-series prediction.
Proposed Method
In the case of the training data, 59 values from the beginning to the end of the total 60 values were used as input data and 59 values from the second to the last were used as the output data. In the input layer, unlike many to one model, 59 values of data were entered as input for each time step to make it easy to grasp the time series of the entire pattern. Similarly, in the Bi LSTM layer, the inherent features of the entire time series were extracted. As these features were put into the LSTM for each time step, each predicted the next time point producing multiple outputs.
Time aware and Feature similarity Self Attention for vessel fuel consumption prediction
Objective
The goal of the research is to predict Fuel Oil Consumption (FOC) of ships. We investigate the ship data and extract three essential ship data properties. We propose the attention mechanisms to represent ship data properties, achieving outstanding prediction performance.
Data
We used 2.5 million amounts of ship and weather data from 19 types of containers. The data were collected by the container sensors from 2016 to 2019. We used ship spec data from Lloyd List Intelligence. The nominal twenty-foot equivalent unit TEU of containers is from 4000 to 13000. A dependent variable, fuel oil consumption (FOC), was acquired from the main diesel engine of the containers.
Related Work
The previous methods for FOC prediction adopted a typical Multi-Layer Perceptron or Long Short-Term Memory, which lacks representing ship data properties.
[3] Panapakidis, Ioannis, Vasiliki-Marianna Sourtzi, and Athanasios Dagoumas. “Forecasting the fuel consumption of passenger ships with a combination of shallow and deep learning.” Electronics 9.5 (2020): 776.
Proposed Method
The proposed model is based on sequence models to represent the sequential property of ship data.
Time aware attention considers irregular time steps in the sequence and emphasizes data depending on the time information.
Feature similarity attention extracts important feature and their weights based on feature similarity in the sequence.
By fusing the models considering time and feature, respectively, the proposed model can represent all the ship properties and achieve outstanding performance.
CRFormer: Complementary Reliability perspective Transformer for automotive components reliability prediction based on claim data
Objective
Reliability prediction in the automotive industry aims to predict the failure of automotive components. Previous studies showed unsatisfactory prediction results when short-term inputs are given. We propose Complementary Reliability perspective Transformer (CRFormer) based on Transformer encoder to achieve enriched representations from a short-term input sequence.
Data
We use the claim data provided by an automobile company. The claim data were collected for 16 years, comprising 951,170 claims, from 2006 to 2021. The information for claim data is as below.
Related Work
InFormer [2] and AutoFormer [3] are Transformer-based time-series prediction methods.
InFormer proposed ProbSparse Self-attention and AutoFormer proposed Auto-correlation attention.
[3] Wu, Haixu, et al. “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.” Advances in Neural Information Processing Systems 34 (2021): 22419-22430.
Proposed Method
CRFormer is based on Transformer to represent sequential information of claim data. From the claim data, a nested sequence that contains both time and mileage-based information is generated and claim features are extracted. The nested sequence embedding module extracts sequential representations of the target reliability, time or mileage, from the nested sequence. Transformer encoder emphasizes the target sequential information. Finally, CA and a prediction layer merge claim features and each sequence step to predict future claims within the remaining warranty periods.
CRFormer achieved competitive performance on reliability prediction for both time and mileage perspectives. The prediction results of the proposed models capture the real claim patterns better, especially in the decreasing and increasing patterns, compared to other Transformer-based models. Regarding increasing patterns in the mileage reliability prediction, CRFormer-F can even capture the increasing patterns, whereas the others can not react at all. The poor performance of the other models in increasing patterns was also revealed through the low f1 scores.
WAY: Estimation of Vessel Destination in Worldwide AIS Trajectory
Objective
To address the problem of destination estimation ahead of vessel arrival, the objective of learning from a data-driven approach is to maximize the conditional probability of the destination port, given the departure and the ordered sequence of AIS messages to the current time.
Data
Satellite AIS data collected from ORBCOMM and port identification data provided by SeaVantage were used during this study. Experiments were conducted on 5-year (Jan. 2016 – Nov. 2020) accumulated real-world AIS data of 5,103 individual ships, where the vessels belong to one of three types: tanker, container, and bulk. A total of ≃ 130K trajectories comprising ≃ 17M AIS messages were mapped between 3,243 worldwide port destinations. A destination-wise stratified split was used to generate the training/validation/test trajectory dataset as 70%, 15%, and 15% of the total.
Related Work
[1] To tackle the data bias issue arising from the irregular interval, the sequence of coordinates is tokenized in units of the spatial grid (1×1miles/grid). [2] Exploits LSTM-based sequence-to-sequence structure, with attention mechanism to predict a future trajectory sequence. [3] Using 4-hot vector representation, where each embedding corresponds to the range of longitude, latitude, sog, and cog, the transformer-decoder architecture predicts the future 4-hot vectors autoregressively.
[2] S. Capobianco, L. M. Millefiori, N. Forti, P. Braca, and P. Willett, “Deep learning methods for vessel trajectory prediction based on recurrent neural networks,” IEEE Transactions on Aerospace and Electronic Systems, vol. 57, no. 6, pp. 4329–4346, 2021.
[3] D. Nguyen and R. Fablet, “TrAISformer-A generative transformer for AIS trajectory prediction,” arXiv preprint arXiv:2109.03958, 2021.
Proposed Method
We first propose to recast the objective formulation defined on the message-wise trajectory processing to the format of a nested sequence. Since it adopts uniform grid-wise processing, such a format mitigates the spatiotemporal data bias, arising along the trajectory progression by irregular AIS collection intervals. Also, the nested structure preserves detailed information on trajectory progression within the unit area.
This work next introduces a novel deep-learning architecture (WAY) and a task-specialized learning technique to fit the redefined objective. Based on a multi-channel representation, the proposed architecture, WAY, processes AIS trajectories in spatial grid steps without losing the detailed AIS properties such as global-spatial identities, local patterns, semantics, and time-irregular progression of trajectory. As a result, the highlight of WAY are the enriching of the details and effective aggregation of representations while processing the AIS trajectory data.
WAY achieves a remarkable improvement compared to the other benchmark models, and WAY with Gradient Dropout (GD) learning technique further demonstrates a performance gain. As well as the overall scores, WAY improves the performance of destination estimation for all steps along the trajectory, especially of which steps were included before the 1st quartile progression. This suggests the importance of enriching the detailed representations of the AIS trajectory and aggregating information effectively.
To generalize the advantage of the task-specialized learning technique in the trajectories with large length deviations, GDs were applied to all benchmark models. Applying GD always achieves a performance gain regardless of the backbone model. The average gains are +1.01% for the overall accuracy and +1.92% for the F1-score.
