Problem with big data (?) during computation of sequence distances using TraMineR
I never saw this error code before, but it might well be due to your high number of sequences. There are at least two things you can try to do:
- use the argument
"full.matrix=FALSE"
in seqdist (see help page). It will compute only the lower triangular matrix and return a "dist" object that can be used directly in thehclust
function. - You can aggregate identical sequences (you only have 12626 distinct sequences instead of 57160 sequences), compute the distances, cluster the sequences using weights (that are computed according to the number of times each distinct sequence appears in the dataset) and then add the clustering back to your original dataset. This can be made quite easily using the
WeightedCluster
library. The first appendix of the WeightedCluster Manual provides a step by step guide to do that (the procedure is also described on the webpage http://mephisto.unige.ch/weightedcluster).
Hope this helps.
An easy solution which often works well is to analyze a sample only of your data. For instance
employdat.sts <- employdat.sts[sample(nrow(employdat.sts),5000),]
would extract a random sample of 5000 sequences. Exploring such an important sample should be largely sufficient to find out the characteristics of your sequences, including their diversity.
To improve representativeness, you can even resort to some stratified sampling (e.g., by first or last state, or by some covariates available in your data set). Since you have the original data set at hand, you can fully control the random sampling design.
Update
If clustering is the objective and you need a cluster membership for each individual sequence see https://stackoverflow.com/a/63037549/1586731