Two Recent Regularization Frameworks for Computer Vision Applications
4 Video Views·Sep 5, 2023
Two Recent Regularization Frameworks for Computer Vision Applications
Bingbing Ni, ADSC
So thank you for coming, we have a large audience today.
So I'm Bingbing Yi, I'm a researcher at Advanced Digital Science Center in Singapore.
And I work with Professor Pierre Molin at UIUC Big Ben.
So today my talk is Two Recent Regularization Frameworks for Computer Vision Applications.
So actually these two topics are quite self-contained, but I just combined in this talk.
So the first talk is Sparse Feature Selection and Grouping Regularization for Image Spatial
Contexts.
And our contribution is we propose Omni-Range Spatial Contexts Modeling Method for Visual
Classification.
And we apply this technique on Fast Food Image Classification and National Flag Image Classification.
So to start with, I give a very short introduction to some state-of-the-art image representation
method.
The name is Bag of Words, which is quite popular nowadays for image retrieval, image classification,
and also some video analysis.
So the input is the image, and we do feature extraction.
So for features, so for example, we can use SIFT features or any other kind of interest
point detectors.
And we have this set of feature points, and we generate a codebook.
You can use any kind of unsupervised clustering methods, such as k-means.
And we have this codebook.
And for every incoming image, we encode it using this codebook by nearest neighbor mapping.
So after that, we got a Bag of Words representation.
It's actually similar with some histogramming.
So this is a very traditional way of how to represent an image, but people later found
that this kind of first-order method does not have so much discriminative capability.
So because the image has very apparent structural emotions, such as the pairwise features, the
relationship between the feature pairs.
So based on this observation, people proposed to use a second-order method to do the image
modeling.
So very commonly, they're using the co-occurrence-based image representation.
So the input is still the image, and we extract the feature points like what we do in Bag
of Words method.
And instead, I will do a direct vector quantization for all the features.
We actually count the co-occurrence frequencies for every pair of features.
So then, if we say we encode the pairwise features, there is always some problem.
So the problem is how to select the neighborhood to define the feature pairs.
So because the feature pairs could be nearby to each other, or they can be as far as the
possible.
So a lot of people have proposed lots of methods.
So I just show some very common choices.
So maybe the first choice is for every feature point, just to select its nearest neighbor
to encode its pairwise information.
And of course, we can have another way.
We can define distance ranges, and within that ball, you will consider that pairwise
relationship.
And if outside of the ball, you just ignore the pieces pair.
So but this kind of method all has problems.
So here is some example.
So you can see this.
I hope you already had your lunch.
So you can see this example.
So sometimes the near context is important, but sometimes the far context is equally important.
So what do we choose?
We choose far context or near context.
I don't think there is some guaranteed optimal way to do that.
So then the problem is how to choose those neighborhood parameters, such as the nearest
neighbor, the k value, and also the distance range value, r.
So maybe one method is that you just divide this parameter space into several bins, such
as, for example, for the r-ball-based method, you can divide different bins equally, and
you can calculate the co-occurrence metrics for every kind of distance parameter r.
But that yields a very high-dimensional feature representation.
So yeah, here is some observations of this kind of method.
So the spatial context configuration space is very large.
If we have the codebook size as k, and if we have r different distance bins, then it
turns out that the feature representation complexity is all k squared and r.
But however, only a small number of configurations can give optimal performance.
And also, if we divide the image representation quality, it's quite sensitive to the selection
of the spatial context configurations, such as the bin size.
For example, if you have two coarse bins, the bin size is too large, then you might
miss some important descriptive information.
But in the other extreme case, if you have two small bin sizes, then your representation
could be overfitting, and that could be even a more serious problem.
How is the performance evaluated?
In terms of the classification accuracy.
We can also see that the spatial context from the adjacent distance bins usually share similar
characteristics, and if we group them, then we can improve the stability and robustness
and compactness of the feature representation.
So that's our motivation.
Our purpose is first we can generate k squared r representations, and then we must have some
method to do the automatic feature grouping and selection, and to yield a final compact
set of features to describe the image.
The problem is how to do this.
We are quite motivated by a recent regularizer.
The name is Octagonal Shrinkage and Clustering Algorithm for Regulation.
So basically, it combines the L1 norm and L infinity norm, and if you look at the constraint
region from this figure, you can see that it actually encourages the grouping of features
and also sparse selection.
So we think that we might use this kind of technique to do our context selection and
grouping.
So here is our formulation based on this regularizer.
Our image is initially represented by a vectorized omni-range co-occurrence matrix, X, which
is of the dimension k squared r.
So our objective is to select and group elements of X, which is to learn the corresponding
coefficient vector W.
So the formulation is actually composed by two terms.
The first term is a discriminative term.
So W, you can use hinge loss or logistic regression.
And for R, it's a selection and grouping regularization term.
So the neighborhood we define here is corresponding to the adjacent spatial bins, so that it encourages
the grouping of adjacent bins if they present similar effects.
So in this formulation, the regularization term is convex but non-smooth.
So we apply Nesterov type smooth approximation to it.
And this approximation has guaranteed approximation accuracy.
So you can find this proof in our CVPR 2012 paper.
And based on this approximation, we apply the accelerated proximal gradient, APG method,
for minimization.
And this optimization method also has a guaranteed convergence rate.
The proof is given in the original paper of APG.
And after we optimize weights, we will remove the context, which according to those zero
entries of W. And we also group the context with the same values.
And after that, we can yield more compact and discriminative representation.
And this new feature representation can be fed into the support vector machine for classification.
So here is our experiment.
So our first experiment is performed on a fast food data set.
So this data set has six categories.
And for each category, we have 18 images.
So three instances for every category.
And we do three-fold cross-validation.
So here are some sample images of this database.
So this figure shows the length coefficient W. So the x-axis actually corresponds to the
visual word pair index.
The y-axis is actually according to the distance being indexed.
And from this result, you can see the very obvious sparse selection and grouping effect.
And we also compare our context modeling method with the state-of-the-art spatial context
modeling methods, which include the spatial pyramid matching, the Markov stationary features,
and the contextualized histogram, and proximity distance kernels, and texture boosting.
And we recently proposed the geometric preserving visual phase.
And we can see that our results yield higher recognition accuracy.
And we also do the sensitivity analysis for our implementation.
So we randomly scale the testing images by zero-mean Gaussian distributed delta scale
values.
So variance of the delta scale values are set as 10 percent, 20 percent, and 30 percent
of the original image scales.
So that means we perform some random scaling of the original images.
And after this random scaling, we apply the previous trained OmniRAN context models on
these distorted testing images.
And then we compare the performance.
And from the results figure, you can show that our method is quite robust to these scaling
changes.
Our second experiment is performed on national flag recognition.
So the database contains 39 countries and totally about 8,000 images.
And for the experiment, we partitioned the data randomly half-half for training and testing.
And we also compared our recognition accuracy with the previous mentioned state-of-the-art
spatial context modeling method.
And our method also gives the highest recognition accuracy.
And here is also a figure which shows some detection results.
So now I'm going to the second part of my talk, which is order-preserving regularization
for time-sequence classification.
So our contribution is temporal order-preserving sparse coding for multidimensional sequence
classification.
So we apply this technique to human action recognition and sign language classification.
So here is our task.
So before, I will give some important notifications.
So our input is a time sequence of length t.
So you can see the y is the input, and it's composed as a sequence of high-dimension vectors
y1 to yt.
And each yi is a d-dimensional vector.
And we also have a set of database sequences, which each sequence has a different length.
And each sequence is corresponding with a different label.
So our objective is to predict the class label for the input sequence y, as shown in this
diagram.
So we take a sparse coding approach to deal with this problem.
So traditionally, first we will stack all training database frames together into a d
by n matrix.
So n is the total number of database frames.
And we perform sparse coding for each input frame individually on the dictionary to obtain
the reconstruction coefficients.
And after we got the reconstruction efficiency for every single input frame, we calculate
the residues of the reconstruction over dictionary sequences belonging to each class.
We sum it up over all input frames and take the class label corresponding to the minimal
sum of residues as our label prediction.
But this type of method has some problems.
Very apparently, for a time sequence input, there exist inter-frame correlations, which
for example is a temporal ordering.
But if you deal with each input frame individually, then that will ignore that important information.
So then it will yield suboptimal recognition performance.
So we think that we need a way for encoding temporal ordering information in this sparse
coding scheme.
So here is our motivation.
So you can see that the left is the input sequence and the right are two dictionary
sequences.
You can see that in the first row, the ordering is quite similar with the input.
But for the second row, the ordering is totally different from the input sequence.
But if you take every frame, you will find that the frames are quite similar.
So our objective is to encourage sparse encoding results which follow the temporal ordering sequence.
So how to do it?
So first we will introduce some important notations.
So we denote alpha ij as a reconstruction coefficient for the input vector i on the
dictionary sequence j, which is a tj-dimensional vector.
And we also define the wj with the same length as the dictionary sequence j, and its value
increases along with the increasing of the temporal index.
So let's say if you have two elements, wjA and wjB, so if A is less than B, then the
waj is also smaller than wjB.
And one natural selection for this kind of w is given in this example, the linear points
on a line.
So given these two notations, the sum of the dot, actually the dot product of wj and aij
can be considered as the expected temporal position of the response for the i's input
vector yi on dictionary sequence j.
So if we give this expected temporal position of the responses, then we can define the below
regressor which, using this regressor, we can penalize the case when the reconstruction
coefficient of the input feature vector i on the j's dictionary sequence temporarily
behind those of the input vector i plus 1 as a timestamp i plus 1 of the input sequence.
We have to say that although this kind of formulation will encourage temporal ordering
preservation, but it's just in an expectation sense, and it does not exactly and restrictly
constrain temporal ordering.
So here is some example if you're using this regressor.
So left is still the input sequence, and the right is two sequences, with the first
one with the right order, and the second one with a different order of the input sequence.
But the individual frames are the same.
You can see using our proposed temporal order preserving regressor, in the first one, this
regression term will not vanish, and for the second result, the value for this regressor
will be large.
So that's intuitive to explain why this regressor encourages temporal order preserving.
So then this is our unifying formulation.
So first is a data reconstruction term, and the second is a group sparsity term, encouraging
the reconstruction coefficients will be within certain dictionary sequences.
And the third is just now we proposed the temporal order preserving term.
And the regression terms are also convex but non-smooth, and we also apply the smooth approximation
to approximate this objective function with guaranteed approximation accuracy, and also
we apply the accelerated approximate gradient method for the optimization.
So after optimization, we got the reconstruction coefficients, and how to do the classification.
So if we denote x-bar j as the set of dictionary sequences from the j-th class, and alpha j
denotes the corresponding reconstruction coefficients for xj, then we can approximate the input
sequence y by using only the optimal coefficients associated with the j-th class, as given in
this form.
And then we do the classification by the minimal reconstruction error, like in this form.
So here is our experiment.
So first we perform some experiment on some synthetic datasets.
For the synthetic datasets, we have database sequences according to these functions, and
we put two sine and cosine to form two-dimensional sequences.
And you can see that for f4 and f5, it's actually symmetric.
And the input sequence we select as x4.
And here it shows reconstruction coefficients if we're using the original sparse coding,
and I'm using the original group sparse coding, and our order-preserving sparse coding.
So you can find that because f4 and f5 are just symmetric, and the value is the same,
so the reconstruction coefficients for them are also symmetric.
And for our method, we can right-select the f4 for our reconstruction.
So here are some experiments on some UCI machine learning datasets.
So we experiment on the UCI Australian Sign Language dataset, which has 2,565 samples.
Each sample is a 15-dimensional sample, and we have subsampled 20 categories for per subset.
And we also test our algorithm on the UCI Spoken Arabic Digit dataset, which contains
more than 8,000 samples and 10 categories, and each sample has 13 dimensions.
And we also apply our method on the CMU motion capture dataset, which has 220 samples,
and each sample is 27 dimensions, and they have five classes.
And the training and testing partition is one by one, and we randomly partition 20 times,
and we actually report the main accuracy and the standard deviation.
So what do you do with this database?
I think this is a quite standard dataset for high-dimensional time sequences.
Is there some advantage you have?
Like, I guess you don't know sign language, and I guess you don't know Arabic.
No, I don't.
Why do you use those?
And actually, frankly speaking, these are the only high-dimensional time sequences datasets
we can find in UCI repository.
So then we compare our results with other state-of-the-art time sequence classification
methods.
And this table gives recognition accuracy in terms of mean accuracy value and standard
deviation average on the 20 runs.
And you can see that our method, which is MTO sparse coding, gives higher recognition
accuracy.
And we also apply our methods, and we evaluate on a recently proposed RGB depth human action
dataset.
This dataset is captured by the Microsoft Kinect, and it contains 12 action categories
under five scenarios, such as in the kitchen, bedroom, living room, office, et cetera.
And four subjects participate in the data collection.
And the feature we use is those joined positions, head positions, hand positions, and some other
motion features, which combine together to form about 459-dimensional feature limitation.
And the testing scheme is a lean one subject out.
And we also compare our method with the previously mentioned time sequence classification method,
and also the two more methods, which is hierarchical maximal entropy Markov model and the
one-level maximal entropy Markov model.
And this table gives a comparison of the recognition accuracy.
And you can also find that our results is higher than the state of art.
And this is the class confusion matrix of the testing results on this human action dataset.
So here comes our conclusion.
So first, we proposed an omni-range spatial context for visual classification, which
yields a compact and discriminative context representation.
And second, we proposed a temporal order-preserving regularizer for time sequence classification,
which improves the classification performance.
And in our future work, we will explore how to use spatial preserving regularization for
image and video analysis.
So questions?
So it seems that your method is much better than the rest.
You mean which work?
Which work?
The second part.
In terms of the accuracy.
Yeah.
I just want to know, what about the training and testing time?
The training and testing time.
So actually for the sparse coding and also our sparse coding method, there's no training.
Because the database is there.
So for every input sequence, then you do that optimization.
So for the testing, it is a little bit longer.
Because for every sample, you need to calculate the reconstruction.
So as a representation.
So I think for every sample in this 400-dimensional sequence, I think for every sample it takes
about 10 seconds on a MATLAB implementation.
But for those small data sets, like the Arabic sign language, the Australian sign language,
Arabic spoken digit, every sample might take one or two minutes.
One or two seconds.
It's smaller.
Much smaller.
Sorry?
Can you speak louder?
Oh, yeah.
Maybe there's something wrong, because it says L2Norm.
I think...
No, no.
Actually, it's L1 2Norm.
Within each database segment, the database sequence, it's a 2Norm.
But it's 1Norm crossing the whole database.
That's why we want to encourage the reconstruction will only happen in certain database sequences.
How do you define the sequence?
The length of the sequence is given by the data set, right?
But for every sequence, the length is different.
And you mean for different sequences, the dimension of alpha is different?
Of course.
We don't constrain the fixed length sequence.
Thank you.
Thank you for coming.

